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5 Field of the Invention 

The present invention relates generally to image coding, and more particularly to the 
compression and streaming of scalable and content-based, randomly accessible digital 
still images. 

10 Background of the Invention 

With the rapid growth of the internet and multimedia applications, there is a great 
demand for new image coding tools that that will provide for high quality processing 
capability, an efficient internal architecture, and flexibility in terms of future 
technological advances. This is a challenge that has been put forth before the JPEG 
15 2000 Committee. Although the main focus should be to provide state-of-the-art 
compression performance, JPEG 2000 should also offer unprecedented content based 
accessibility of the compressed format to support applications of various needs. It is 
highly preferred and advantageous that image content be accessed, manipulated, 
exchanged, and stored in compact form. 

20 In order for JPEG 2000 to be the standard coding foundation of new generation image 
processing systems, it must provide for efficiency in coding, different types of still 
images (bi-level, gray-level, color) with different characteristics (natural images, 
scientific, medical imagery, rendered graphics, text, etc.) within a unified system. In 
addition to providing low bit rate operation with quality performance superiority to 

25 existing standards, this new system should include many modem features as listed in 
the JPEG 2000 requirement document. 

The open architecture and the set of algorithms presented in this document are based 
on Digital Accelerator Corporation's (DAC) Region based Image Coding System 
(RICS). R1CS has not only achieved a rate distortion performance competitive with 
30 best known compression techniques, but also demonstrates a high degree of openness 
and flexibility that will accommodate most well known algorithms as well as new yet 
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10 be implemented. The features supported by RICS covers almost all those listed in 
JPEG 2000 Requirements document. 



Generality 

Generality is a primary concern in the architectural design of RICS. The RICS 
5 architecture attempts to be an integrated platform that supports and facilitates a 
variety of applications that may have different characteristics and requirements. For 
example, providing efficient lossless and lossy compression in a single code stream; 
efficient processing of compound documents containing both bi-level (text) and color 
imagery; progressive transmission by pixel accuracy and/or by resolution; random 
10 access of arbitrary shaped regions; and so on. 

Openness 

We also understand that the new JPEG 2000 standard is intended to be a dynamic, 
rather than static, suite of coding tools that support the new generation imagery 
applications and, at the same time, keeping abreast with the progress of technology. 
15 The mathematical foundation for those leading candidates of new image coding 
methods (most noticeably the various types of multi-resolution analysis techniques, 
such as wavelet transforms) is relatively young and still under intensive investigation. 
If an architectural design is based on or restricted to one or several particular existing 
coding methods, it may become outdated very quickly. 

20 In attempting to produce a flexible and open platform, the RICS architecture 
organizes the image coding process into a set of functionally separable modules, such 
that each module can be developed and optimized individually. Furthermore, all 
modules in the system are designed to be functionally orthogonal with each other, in 
the sense that a new algorithm, without affecting the functionality of other modules, 

25 can effectively replace the base algorithm of any specific module. 

This open architecture will not only be able to accommodate future new algorithms, 
but also makes compatibility with other standards an easy and natural extension of 
many concepts, as will be explained later in this document. 



2 



WO 00/49573 PCT/CA 00/00 134 

Accessibility 

Content based accessibility is becoming an important feature in supporting 
applications such as multimedia database query, internet server-client interaction, 
content production, remote diagnostics, and interactive entertainment. The coment- 
5 based accessibility requires that semantically meaningful visual objects be used as 
basis for image data representation, explanation, manipulation, and exchange. With 
images being represented in compressed format, it is desirable to perform retrieval 
operations directly in the code space without requiring image reconstruction. In fact, 
any search algorithms that require image reconstruction will be infeasible from a 
10 practical viewpoint because of the huge amount of images in most image databases. 
The previous JPEG and many existing coding techniques focus primarily on the issue 
of compression ratio, paying minima] attention to the need of content based image 
retrieval. 

RICS, by its name, has a fundamental consideration to various types of regions in 
15 images. The regions referred to as ROl (Region of Interest) are usually user-specified 
primitive geometric shapes, such as rectangles or circles. The regions that define the 
visual objects are usually of arbitrary shapes. Regions can also be generated as the 
result of certain mathematical operations or transform properties (e.g. significance of 
transformation coefficients) used to partition the image into disjoint regions of various 
20 (e.g. tiles, hierarchical blocks, or arbitrary shapes). 

In designing the RICS system, DAC considered carefully the distinction between the 
concept of an object and that of a region. RICS is open to very general definition of 
region. A region is a 2D spatial identity with pure syntactic contribution to a code 
stream. In contrast, an object may contain semantic information. Therefore, the 
25 region is perceived as a more elementary and reliable description than the object. 
RICS is region based, not object based. RICS supports a rich set of region types, 
from primitive geometrical shapes, tiles, hierarchical blocks, to most general arbitrary 
shapes. 

The region based coding strategy effectively supports the content-based accessibility 
30 of imagery data. Specifically, this ability enables random access to the code stream as 
well as providing a processing channel for user defined regions of interest or tile 
based techniques. Supporting MPEG-4 object based accessibility is one of the main 
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objectives of the RJCS design. Furthermore, region-based coding provides a natural 
bridge for 'transcodability' with JPEG and JBIG. 

Scalability 

Many applications require image data that is available at different resolutions or 
5 qualities for decoding. For example, in a progressive transmission process, the bit 
rate control mechanism should allow the image data to be transmitted in certain 
priority order, and be able to truncate the remaining data flexibly, either upon request 
from the receiving terminal or upon channel limitation. Scalable image coding 
involves generating a coded representation (code stream) in a manner that facilitates 
10 the reconstruction of the image at multiple resolutions or qualify levels by scalable 
coding. 

Ideally, the control of scalability should be centralized in a single module as the last 
stage on the encoder side right before the code stream is fed to the communication 
channel. Furthermore, it is desirable that the scalability control module can 
15 completely handle the required processing locally, without any further request 
propagating back to any previous stages in the encoder. In this way the scalability 
control module avoids the need for multi-pass computation. 

The RICS architecture is designed to support three types of scalability: scalability in 
terms of pixel precision, spatial resolution, and regions. 

20 Compactness 

There is no doubt that the new image coding standard should offer a higher 
compression performance than the former JPEG, especially at the low bit rate end. 

Integrating the compactness, scalability, and accessibility into a general purpose, 
flexible, and open architecture represents a challenge for JPEG 2000. The RICS is 
25 designed to provide a solution. The basic idea of region based coding is as follows: 

* The input image data, after certain transformation, becomes a set of image 
primitives. This set of primitives can be wavelet transform coefficients, DCT 
coefficients, other transform coefficients, or even raw image data. 
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• The image primitives are grouped into regions. Region definition can come from 
user defined ROI, from other application modules, or from certain automatic 
segmentation algorithms running in the primitive space. 

• Each region contains one or more independent coding units (1CU). The primitives 
5 in an ICU are encoded and decoded independently, without reference to primitives 

of any other ICU. This procedure is called the intra-region coding. The outcome 
of an ICU operation is a code block. 

• A multiplexer (MUX) is employed to integrate the code blocks into the final code 
stream. 

10 A Brief Description of the Drawings 

Figure 1 is a simplified RICS block diagram. 
Figure 2 is a detailed RICS block diagram. 
Figure 3 is subband decomposition schemes. 
Figure 4 is a Wavelet Filter Library. 
15 Figure 5 is a performance comparison of YUV and standardized color systems. 
Figure 6 is geometric shapes supported by RICS. 
Figure 7 is a hierarchical partitioning. 
Figure 8 is the relationship between regions and objects. 

Figure 9 is threshold masks obtained from Level 1 Lena Wavelet Decomposition. 
20 Figure 10 is a Level I common mask obtained by thresholding the combined data set. 

Figure 1 3 is individual region masks obtained by separating raw the common mask. 

Figure 12 is individual region masks obtained by separating raw common mask. 

Figure 13 is a spectra! magnitude of zigzag spectrum as a function of position. 

Figure 14 is common mask spectrum filter sizes and captured coefficient numbers. 
25 Figure 15 is a coefficient banding concept used to quantize the common mask 

spectrum. 

Figure 16 is Quantization Band Sizes for Common Mask Spectrum of size 128 by 
128. 

Figure 17 is Spectral Quantization Band Sizes for Other Common Mask Sizes. 
30 Figure 18 is a Mask Overhead for Grayscale Images. 

Figure 19 is a low pass Common Mask Approximation for general coefficient 
classification. 

5 
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Figure 20 is a Mask Quantization following the Spectral Content for Bit Allocation. 
Figure 21 is Translated Common Masks for Remaining Resolution Levels. 
Figure 22 is the three types of ICU structures. 
Figure 23 is a Pavement of a region using iype-1 ICUs. 
5 Figure 24 is a Embedded quad tree flowchart. 
Figure 25 is VLC Codes for EQW. 
Figure 26 is the transcode with JBIG. 

Figure 27 is the 'trans-out' and 'trans-in* modes for transcoding with JBIG. 

Figure 28 is Wavelet Mallot Decomposition for Lossy Color Transform Data. 
10 Figure 29 is Level Priority Processing Order for Each Channel (Lossy Case). 

Figure 30 is Level Priority Processing Order (Lossy). 

Figure 31 is Level Priority Color Interleave Processing Order (Lossy). 

Figure 32 is Level Priority Color Processing Order (Lossless). 

Figure 33 is Level Priority Color Interleave Processing Order (Lossless). 
15 Figure 34 is Typical MUX List Data Structure. 

Figure 35 is SNR Progressive Bit Budget Distribution Scheme. 
Figure 36 is MUX List Overhead for Y-Channel 8-Level Decomposition. 
Figure 37 is MUX List Overhead for Square Images of Various Sizes. 
Figure 38 is MUX List Overhead for Square Images of Various Sizes. 
20 Figure 39 is General Region Level Priority Color Processing Order (Lossy). 
Figure 40 is Region Level Priority Color Processing Order (Lossy). 
Figure 41 is Color Interleave Region Level Priority Processing Order (Lossy). 
Figure 42 is Region Level Priority Color Processing Order (Lossless). 
Figure 43 is Transparent Region Level Priority Color Processing Order (Lossy). 
25 Figure 44 is Transparent Region Level Priority Color Processing Order (Lossless). 
Figure 45 is a graph representation of the transparent region level in color 
mode for the 4 DCT region channels. 

Figure 46 is a graph representation of the region priority level in color 
mode for the 4 DCT channels. 
30 Figure 47 is a graph representation of the absolute region priority level 
color mode over the 4 DCT region channels. 

Figure 48 is a graph representation of scaled region priority level color 
mode of the 4 DCT channels. 

Figure 49 is a graph representation of the region percentage priority level 
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in color mode over the 4 DCT region channels. 

Figure 50 is a graph representation of the mixed processing multiplexer mode 
of operation over the 4 DCT channels 
Figure 51 is a diagram of the basic lossless header structure. 
5 Figure 52 is a diagram of the basic lossy header structure. 
Figure 53 is a diagram of the basic header tag structure. 
Figure 54 is a block diagram of the basic bit stream syntax for normal modes 
of operation in the lossy case. 

Figure 55 is a block diagram of basic bit stream syntax for region modes of 
10 operation. 

Figure 56 is a block diagram of the basic bit stream syntax for mixed modes 
of operation. 

Figure 57 is a block diagram of the JPEG transcode/decode method 
Figure 58 is a block diagram of the JBIG transcode/decode method 
15 Figure 59 is a block diagram of the modified post filtering procedure 
Figure 60 are three gradations of post processing in digital still images- 
Figure 61 is a representation Edge areas where the de-ringing filtering is 
selectively applies. 

Figure 62 is a table of Test flmage: Camera (256x256 grayscale) Test Image: 
20 hk (256x256 color) 

Figure 63 is a table of Test Image hk.raw results 

Figure 64 is a table of Test Image camera.raw (grayscale) results 

Figure 65 is a table of Test Image hk.raw (color image)results 

Figure 66 is a table of Camera raw (part a) results 
25 Figure 67 is a table of Camera.raw (part a) results 

Figure 68 is a table of HK raw results 

Figure 69 is a table of: Quality versus iterations. 

Figure 70 is a table of: Quality versus iterations. As discussed. 
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A Detailed Description of the Drawings 
The System Architecture 

The architecture of the RICS system can be described as the scheme of dynamic 
5 screaming of code blocks. In this design, the image primitives of ICUs generate unit 
code blocks. A code block is a logically independent unit of coding and decoding 
which does not rely on information contained in any other blocks. Compression is 
achieved in each block coder, and the coding efficiency of the block coders 
determines the overall efficiency of a RICS system. The openness of the system is 

10 reflected in the different coding algorithms that can be used to produce different code 
blocks. The system uses a multiplex structure (a MUX) to assemble the code blocks 
into the code stream and realize the bit rate allocation. In short, the RICS architecture 
allows for flexibility in the areas of compactness and openness to the block coders 
and, at the same time, allowing the multiplexer to handle the various schemes of 

15 scalability and random accessibility to the code stream. 

It should be noted that the block used in RICS is a logical unit rather than a geometric 
concept. A code block may correspond to an 8x8 tile (in the case of JPEG mode 
coding), a pyramid data structure in zero tree like coding schemes, a rectangular area 
in block based coding schemes, or an arbitrary shaped area in the raw image buffer. 
20 The independence of encoding and decoding is the primary requirement of a code 
block. A code block is the outcome of an intra-region coding. 

Figure 1 illustrates the simplified RICS coding architecture. A more detailed diagram 
is shown in Figure 2. A detailed description of the function of each module is given 
25 in this document. Because the RICS is intended to be an open architecture, DAC does 
not limit each module to any specific algorithm. Instead, any individual algorithm 
can be placed into a module as long as it meets the functional requirement of that 
module. Supporting algorithms are included for certain modules, which reflect 
preferred operation of the overall system. 
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Types of Input Image 

As shown in Figure 2, the R1CS architecture supports ihe coding of three types of 
image data: grayscale, color, and bi-level images. 

Transformations 

5 In a typical multi-resolution coding scheme, an image is transformed via a multi- 
resolution decomposition process. In the proposed architecture, transforms such as 
KL, wavelet, wavelet package, lifting scheme, etc. can be placed in the transformation 
module. These transforms produce a set of decomposition coefficients {Cij} at 
different resolution levels and in different spatial orientations. 

10 The RICS architecture also supports DCT or windowed Fourier transform as a 
transformation technique. This is mainly for the transcodability with JPEG. It should 
be noted that the Fourier based transforms have been studied for more than a century; 
its theory is relatively complete and its mathematical and physical properties are well 
understood. Particularly, its translation, scaling, and rotation properties may be very 

15 useful for content based retrieval computations. On the other hand, wavelet 
transforms are relatively new, and many of its properties require further 
investigations. As a result, the support of DCT may have an impact beyond the sole 
backward compatibility consideration. 

RICS also allows the NULL transformation (that is, no transformation is applied at 
20 all). In this case, an identity transformation is applied to the raw image data as the 
transformation step. The NULL transformation is useful in several instances. For 
example, it is usually not beneficial to apply DCT or wavelet transforms to bi-level 
images (text) for compression purpose. Another example is the residual images in 
video coding. The information in a residual image is the difference between video 
25 blocks and has a high frequency spectral content already. It is highly questionable 
whether it is beneficial at all to apply another mathematical decomposition (DCT or 
wavelet) to this type of data for the purpose of compact coding. 

Region Definition 

The function of this module is to partition the coefficients produced by the 
30 transformation module into a number of spatial regions. The RICS supports three 
region schemes. 
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1 . Automatic partition based on the preordering of the coefficients. 

2. Partition based on user defined ROI or object related shapes. 

3. Partition based on tiling. 

The first partition scheme is also referred to as the region hierarchy formation process. 
5 Consider the transformation coefficients as a set. The region hierarchy formation 
process partitions the set into a number of hierarchically disjoint subsets according to 
certain definitions of significance. In providing a very general partitioning technique 
captured in a very general region based control architecture, RICS can perform highly 
flexible progressive transmission modes of operation that depend on data priorities set 
10 up for the code stream. 

Scheme 2 deals with user specified ROI, typically primitive geometrical shapes, such 
as rectangles or circles, as well as object related shapes. Object related shapes could 
come from a variety of sources, such as user input, motion analysis in video 
compression, etc. 

15 Scheme 3 is a simple partition and requires minimal for shape coding. This scheme is 
essentia] in the JPEG mode. Some wavelet based compression techniques utilize this 
scheme to explore coding efficiency. This scheme offers very little support for 
content based accessibility. 

Hierarchically disjoint regions can be used in combination with user defined ROI in 
20 still image processing and objects in video processing. However providing a fair user 
partition for detail information is difficult in still image compression. But automatic 
partitioning or preordering techniques can be performed to control user selections in 
both arbitrary and block based modes of operation. The research presented in this 
document introduces a new multi-level control architecture for advanced multi-level 
25 image processing. The ability to perform a host of partitioning and preordering 
techniques in both normal and region based modes of operation is encompassed in 
this architecture. Both arbitrary and automatic region formation schemes are handled 
in the same manner at a high level. 

Region Shape Coding 

30 Three types of region shape are supported in RICS: tiling, primitive geometric shapes, 
and arbitrary shapes. 

10 
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1. Tiling requires only a small set of parameters to describe the configuration 
especially in sequence based processing modes. However the sequences can be 
organized and presented in many ways in packing them into the final code stream. 

2. Primitive geometrical shapes can also be coded efficiently. For example, a 
5 recungle can be defined by four integers U y mii) and (xm^ymti , or 

(xmiiv J' mii) and (width, height) etc. 

3. Arbitrary shapes are more difficult and costly to encode. Partitions produced 
automatically by image analysis algorithms may contain many small regions. 
Specifically, the auto-region detection routines presented here produces 

10 hierarchically organized data partitions. This presents a highly challenging 

problem for shape coding. RICS provides two practical solutions to this problem. 
One is a quad tree based hierarchical region description. This mode of operation 
follows bit level ordering at different resolution levels (see embedded quad tree 
wavelet (EQW) in Ch. 4.) The other is a DCT coded multi-level region channel 

15 definition followed by preordering the partition (given the restraints of the code 

stream in terms of overhead). Though the coding efficiency of the second solution 
is currently slightly poorer than the first it does contain a number of attractive 
features: 

• It provides a single representation for multi-level bitmaps. 

20 • From this single representation, region masks can be reproduced at arbitrary 

resolution levels, which is useful for subband coding. 

• It generates curved shapes, which potentially (e.g. in case of large number of 
complex curved regions) could outperform quad tree based region 
descriptions. 

25 • It has several useful properties, such as translation and rotation properties, 

which the quad tree based description does not offer. In fact, a quad tree 
representation will change dramatically if an object is slightly translated in the 
image, making this type of representation not suitable for content based 
retrieval applications. However, the main advantage of the quad tree 

30 representation is its simplicity and most noticeably in block based modes. 
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The region shape code is included in the code stream when the region definition is 
determined at the encoder. In the case of decoder specified regions, the region shape 
coding has a whole new meaning. 

Imra-Region Coding 

The function of the intra-region coding is to arrange the transformation data in an 
arbitrary shaped region into a one dimensional code stream. Regardless of the region 
definition scheme or the region coding technique, this streaming process requires an 
intermediate state where a control architecture can be designed to tailor the region 
channels whether dealing with a bitmap mask, an auto-detection routine or any other 
of numerous classification techniques. At the decoding end, the inverse routine 
generates the same mask to unpack the values from the one dimensional code stream 
and place them back into the correct position in each region. 

Intra-region coding is completed in block coders. Different block coders can be used 
to produce the ID code stream. For JPEG mode, the zigzag scanning/quantization 
routine is called to pack an 8x8 DCT block. For wavelet based coding, both explicit 
quantization and implicit quantization schemes can be used. In particular, embedded 
zero tree and embedded quad tree can be used as implicit quantization schemes. 
Furthermore most implicit quantization schemes are implicitly decodable. In dealing 
with bi-Ievel images, a JBIG routine can be called a block coder. This effect can be 
staged by calling an implicit quantization scheme using one bit plane. Alternatively, 
efficient JBIG routines can be called block coders in an embedded coding scheme at 
each of the multiple bit planes. 

The Multiplexer 

The function of the multiplexer is to assemble the code blocks derived from different 
subbands and different regions in proper orders into a single code stream. Due to the 
richness of region definition and block coding schemes, there are plenty of ways to 
pack the code blocks. Different ways to merging the data lead to different 
transmission priorities. For all transmission ordering modes, the final code stream can 
be transmitted progressively, and can be truncated at any desired place. 

The notion of using a multiplexer has been adopted in some standardized processes 
such as MPEG. In contrast, the use of a multiplexer in the design of a still image 
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coding system is rare. The region based coding strategy opens the opportunity to 
systematically explore the syntactic and semantic richness in code stream ordering 
and transmission medium. With the image segmentation techniques improving in the 
future, region shapes will become more accurate in tracking objects in the image. In 
5 that case, the multiplexer will not only work as a syntactic composer, but also impose 
semantic meanings to the code stream. 

Hiphlipht of Features 

The R1CS is a true open architecture. It supports not only the algorithms DAC has 
developed, but also can accommodate most existing well known compression 
10 algorithms. Users may include their own functions that are appropriate to their 
application in a number of modules such as transformation, region definition, region 
shape coding, and intra-region coding all under the MUX control architecture. It is 
also implicitly flexible to new technological advances. 

• DAC's current implementation offers superior low bit rate performance that is 
15 competitive with best existing compression techniques. 

• The richness in region definition in the RICS allows great accessibility, thus 
providing a solid foundation for content based image applications. 

• It covers compression for both continuous tone and bi-level images in a single 
unified architecture. 

20 • It provides lossy and lossless compression in a single, natural, code stream in the 
course of progressive decoding. 

• It provides a variety of progressive transmission modes that allow images to be 
reconstructed with increasing pixel accuracy or spatial resolution using region 
priority modes for user specified ROl or system defined significant areas. 

25 • It supports both fixed rate and fixed size modes. 

• It supports very flexible random access to and processing for regions with 
arbitrary shape. 

• It provides a graceful backward compatibility (or transcodability) with JPEG. 

• The generic region definition in RICS is a very suitable interface for object-based 
30 coding schemes currently under development by MPEG-4. (In fact, as the 
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arbitrary region shape begins to fit dynamic objects with more accuracy, motion 
estimation is more definite, and consequently allows for more efficient error 
compensation using region-based coding.) 

• Our general region shape definition provides a solid basis for object based 
5 composition and object based information embedding. Multiple objects with 

arbitrary shape are accepted. 

• Our bit stream is robust to bit errors. Each unit structure used by the MUX is 
independently decodable. 

• Incorporating standardized encryption techniques with the RICS system is 
10 straightforward. 

Transforms 

In order to support multiple application needs, provide transcodability, and 
accommodate future growth, the RICS architecture is designed to support three 
15 categories of transformation: the DCT, wavelet transforms, and a special NULL 
transform. 

DCT 

The RICS architecture supports the DCT transforms as defined in the baseline JPEG. 
Wavelet Transform 

20 The RICS architecture supports various kinds of subband decomposition schemes, 
including the three schemes in Figure 3. 

The Null Transform 

In dealing with the compound documents with mixed contents, it is sometimes 
required to encode some areas of the image without any transform. In particular, 
25 regions with bi-level pixels usually do not need to be transformed for coding purpose. 
In coding these regions, the transform stage is bypassed; the effect is simulated by a 
NULL transform stage in the process. 

The present version of RICS uses totally 34 types of wavelet kernels as given. in the 
30 table in Figure 4. In our experiments, the biorthogonal filters (Bior9-15 of the table) 
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seemed 10 give the best compression. Both low pass filtering and high pass filtering 
are done using convolution. After filtering, the wavelet coefficients are down 
sampled by 2. This process is repeated using the low pass part until the desired 
decomposition level is reached. In inverse wavelet transform, quadtrature mirror 

5 filters (QMF) for low and high pass are used. Then the coefficients are up-sampled 
by 2. 

Color Transform 

In addition to YUV format, the RICS system uses the Karhunen-Loeve Transform for 
color transformation. Following the notation of statistics we term this process as the 
10 color standardization. The KL color transform requires more computation than YUV 
transform. Figure 5 shows the PSNR comparison of the two color transforms. More 
test results are also available from DAC 

Repion Definition and Shape Coding 

15 The notion of region processing plays a fundamental role in the operation of RICS. 
The choice of region based coding is motivated by many application needs such as 
content based retrieval, interactive multimedia application, graphics object and image 
composition, and coding of dynamic object in video compression. 
There are two fundamental issues in a region based coding strategy: defining regions 

20 and coding the regions. In the RICS system, region coding is divided into the shape 
coding and the intra-region coding (the content coding). Section 3, describes the 
various schemes for region definition and shape coding. The intra-region coding is 
discussed below. 

From the viewpoint of JPEG 2000, the process of region definition does not have to 
25 be standardized. However, schemes for forming regions that the JPEG 2000 will need 
to support should be defined clearly. The RICS supports three types of region 
definition. 

• Tile based organization 

• primitive geometric representations 
30 • arbitrary shapes 
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Region definition is an optional step: an image can be coded without specifying any 
region (the non-region mode). In this case, the entire image area is considered as a 
single region. 

Tiling: Definitions and Shape Coding 

5 Tiling is perhaps the simplest region definition scheme. In this scheme, the entire 
image area is divided into a number of rectangular blocks. In particular, 8 by 8 tiies 
are used in the JPEG coding mode. Most tiling approaches cannot cover the natural 
shapes, region trends or partial objects in a picture. 

For coding a tiling scheme, a small set of global parameters will be sufficient. The 
10 primary parameter is the size of tiling block and the technology is developed around 
this theme. In case of subband decomposition, tiling block size may vary from one 
subband to another. However, the RICS architecture has a sound operational 
foundation. It can be operated in both tile based or arbitrary processing modes of 
optional arbitrary region formation schemes. 

15 Primitive Geometrical Shapes 

Primitive geometrical shapes are ideal for supporting user interaction; i.e. user 
specified ROI, and for processing compound documents where the text areas can be 
well covered with one or more rectangular boxes. Geometric shapes currently 
supported by RICS are listed in the Table in Figure 6. 



20 
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Arbitrary Regions and a Generic Binary Partition Hierarchy 

Let C 0 be the set of image primitives. Given an ordering relationship ^, generic 
binaries partition of C 0 into C to and C M is defined by the following conditions. 

(1) C lo uC M =C 0 , 

(2) C 10 nC n =0>,and (Eq.III.l) 

(3) for any a e C ]0 and b e C u , a >- b holds. 



Recursively, each of the new generated subsets can be further partitioned into two 
30 subsets, resulting in a hierarchical structure to represent the partition (see Figure 7) 
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A useful ordering relation popular in the compression community is "more significant 
than": a y b whereby a is more significant than b. The definition of significance can 
be very general. One definition may be used to create an ordering, or several 
5 definitions are combined together to generate an ordering. This can be illustrated by 
the following example shown in Figure 8. Suppose the region masks shown in the 
leftmost image are generated by considering the magnitude of wavelet transform 
coefficients as the measure of significance. The region mask in the middle image is 
an object shape generated by another definition of importance that, for example, can 
10 be the intensive dynamic region in a video frame. The rightmost image shows the 
intersection of the two region masks, which is a new region partitioned according to a 
multiple definition of significance. 

As an example, if the significance is measured by magnitude of wavelet transform 
15 coefficients, then the ordering relation is simply the numeric **>** relation. To create a 
binary partition, a set of thresholds is sufficient. The first threshold, say 7J,, separate 
the initial set C 0 into two subsets, C 10 and C n . Subsequent threshold values can be 
specified to recursively partition the new subsets. 

The above partition scheme produces accurately a number of disjoint subsets which 
20 satisfies the strict ordering relationship of >■ . However, it usually produces many 
small, scattered regions. Consequently, the representation of region shapes for these 
subsets can be potentially expensive. From the representation point of view, fewer 
numbers of smoothly shaped regions are desirable. In the following, a less strict 
partition scheme is defined, which takes the above partition scheme as a special case. 

25 Instead of requiring the condition a>b to be held for all aeC I0 , beC lt , it is 
replaced by a looser condition 

30 -J? e C, , b >■ a , for any a e C 0 . (Eq.III.2) 

Intuitively speaking, this definition defines a partition in which no element in C, 
proceeds any element of C 0 . In other words, some elements belonging to C, (by 
Eq.III.l) may now be placed in C 0 , but, no element belonging to C 0 C, (by Eq.III.l) 
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can be placed in C, (Eq.III.2). In doing so. the subset C 0 will be enlarged, and the 
region shape of C 0 will be smoother. At the same time, the enlargement of C 0 is 
controlled to such an extent that the loss of classification accuracy is limited to a 
reasonably low level. 

5 Another scheme, which is symmetric to the Eq.III.2, can be defined similarly for the 
following condition. 

10 e C 0 , a -< b . for any b e C, . (Eq.IJI.3) 

In this case, we ensure that no element in C 0 is proceeded by any element in C,. 

It should be poinied out that there is a tradeoff between the enlargement of C 0 and the 
space requirement for the region shape representation. It is possible (and indeed quite 
15 often) that there are some isolated scattered points that belong toC 0 . But, in order to 
exclude those isolated points from C J( we have to sacrifice many elements from C t . 
This will result in a very large C 0 , a degraded classification. An approximation to the 
scheme of Eq.III.2 is introduced to rectify this problem. A small number of elements 
in C 0 are allowed to be "misciassified" into C,. Those misclassified elements can be 
20 separated later in the intra-region coding stage that generally follows natural 
processing orders available for further classification of the partitioned elements. 

When such region hierarchy formation schemes are applied to image primitive planes, 
they serve as preordering processes that set up a partial ordering among the 
primitives. Then, the problem of coding this ordering becomes a geometry problem 
25 for coding the shapes of the partitioned regions. Transformations such as wavelet are 
known to use a joint spatial/frequency domain methodology. The region hierarchy 
formation scheme mentioned above has an intuitive geometric explanation for this 
class. Transformations such as DCT and Fourier transform are purely frequency 
domain methods. It is interesting to note that when the above partitioning scheme is 
30 applied to a 2D DCT transformation plane, the generated regions are roughly 
circularly shaped bands. The popular zigzag sequence used in JPEG turns out to be a 
simple and reasonable approximation of these bands. Therefore, the region based 
coding strategy accommodates naturally both spatial/frequency domains and pure 
frequency domain transformations. 
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Automatic Region Formation in the Wavelet Transform Domain 



DAC has developed an automatic region formation scheme that is used to categorize 
wavelet coefficients in the transform domain according to magnitude. The procedure 
will be outlined in the following sections. Also included are many of the design 
5 details used to develop the first version of the general region classification scheme. 
The advantages and disadvantages of the technique as well as performance and 
overhead issues will be discussed. Finally conclusions will be drawn to summarize 
the results. 

Detecting the Regions 

10 The wavelet transform is used to decompose the original image information into a 
multi-resolution hierarchy of data that is more suitable for compression. The result of 
completing one pass of the 2D wavelet transform is one low pass part (LLi) and three 
high pass parts (HLj, LHj and HHj.) The transformation procedure is repeated using 
the low pass LL pan as the starting point for each succeeding pass. The LL part is an 

15 approximation of the original image at a lower resolution. At each resolution level 
the HL parts carry the vertical, the LH parts carry the horizontal and the HH parts 
carry the diagonal edge information. The level of detail information contained in any 
particular orientation is specific to the local response generated in the choice of 
wavelet kernel. The original image is transformed into a unique band pass response 

20 hierarchy of detail information. In any compression process, it is the highest energy 
coefficients that are considered first for making a contribution to the reconstructed 
image. It is this transformation property that is used to partition the coefficients into 
regions of interest corresponding to high energy locations in the original image space. 

The magnitude of the coefficients in a particular subband is related to the response 
25 that the original image information has to the transform kernel. Sharp edges in the 
original image (or a lower resolution LL part) are indicated by an increased kernel 
response in terms of coefficient magnitude, in the general vicinity of a specified area 
of concern. Other image changes that are not as abrupt will translate into a gradual 
response that is less significant. In those areas where little or no change occurs, the 
30 wavelet transform will indicate little or no response. 

Level I Kernel Response Threshold Images 
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DAC's automatic region formation scheme makes use of coefficient magnitude 
information to categorize the data. The starting point for the procedure is at the 
highest resolution level of the wavelet transform hierarchy looking at the three detail 
data sets (HLi, LHj. HHj.) A 2 bit representation of each detail orientation for a 256 
5 by 256 Lena image is given in Figure 9. These images are obtained by using 

threshold values to categorize the coefficients by magnitude. The wavelet kernel used 
in this case is the standard 9-7 filter set implemented in a lifting scheme. 

A decaying histogram procedure is used to determine the threshold values in each 
case. In this particular analysis the coefficients are partitioned into regions according 

10 to decreasing levels of magnitude; 10% of coefficients are partitioned into region 1, 
15% into region 2, 25% into region 3 and 50% into region 4. Note that the largest 
coefficients appear darker in the images while the smallest appear lighter. The 
threshold values are calculated automatically for each orientation. The values used to 
generate the partition in each case are given in Eq.III.4 (assume Q is the magnitude of 

15 the coefficient under consideration.) 



HL,: 0 S C; < 2, 2 < Q < 4, 4 < C ; < 8 and Ci > 8 

LH,: 0 < Q < 3, 3 < Q < 5, 5 < Q < 14 and Q £ 14 (Eq.III.4) 

HHi: 0 £ Q < 2, 2 < Q < 3. 3 < Q < 5 and C t > 5 



25 

The set of threshold images appearing in Figure 9 is motivational in determining a 
region formation scheme based on this approach. It is apparent that wavelet 
coefficients can be classified in this manner to form a mask. However, the large 
overhead required to code each individual mask must be solved in order to make this 
30 type of classification scheme viable for implementation. 

Formation of the Raw Common Mask 

The first step taken in the development of a technique designed to reduce the mask 
overhead is to consider a common mask approach that can be used to capture the most 
important coefficients between all three orientations at the lowest resolution level. 
35 Given that the data range is different in each orientation, the data is normalized to the 
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largest range in an absolute value sense. This step is taken in order to compensate for 
the range differences that exist between the data sets, and to ensure that corresponding 
pixels from each orientation can make an equal contribution in the formation of the 
common mask. This step is illustrated in Eq.III.5. 

HL, = NewRangetHL,) = ScaleRange(MaxRange(LH„ HL, , HH,)) 

LH, = NewRange(LH,) = ScaleRange(MaxRange<LH„ HL, . HH,)) (Eq.111.5) 

HH i = NewRange(HH,) = ScaleRange(MaxRange(LH,, HL, , HH,)) 



Once the data ranges for the smallest two sets have been scaled to that of the largest, 
the common mask values can be determined. The next step is to generate a new 
normalized data set by taking the maximum value between scaled orientations at each 
15 pixel location. Let C (i) be the new coefficient obtained in this procedure and let 
C hli . C lhi and C hhi be the individual scaled values from each orientation. The 
new data set is formed using the step illustrated in Eq.III.6. 

C (i) = MAX(ABS(C HLi(0h ABS(C ' HL i(i)). ABS(C H u(i))) (Eq.III.6) 

20 The final step in forming the common mask is to threshold the new coefficients based 
on the histogram decay and the percentage of the total data amount to encapsulate in 
each region category. The same values mentioned earlier are used in this case (10%, 
15%. 25% and 50%.) The raw common mask categorization image appears in Figure 
10. The threshold values used to generate the mask are given in Eq.II1.7. 

H'i: 0 < Q < 5, 5 < C- < 11, 11 < C, < 26 and Q > 26 (Eq.III.7) 



30 Note the level of detail contained in this image. High frequency edge information 
representing the most important coefficients appears in the darkest areas. Low 
frequency changes representing areas of the image that are relatively uniform appear 
in the lightest areas. The combined kernel response from each orientation follows the 
high energy areas that exist in the original image. 
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Common Mask Conditioning Using the DCT Method 

Generally, the raw size overhead of the common mask obtained in the previous 
section is still too large to be included in the final bit stream for most image 
processing needs {i.e. an overhead of 2 Bpp for this particular 4 region mask). This 
5 section outlines the next step in the automatic region formation scheme. The DCT 
transform is used to reduce the overhead size to an acceptable amount. 

The DCT is not applied directly to the common mask data in its mult-valued form. 
Rather it starts by considering each region contained in the common mask as an 
binary data set such that the sum of the individual data sets is equal to the multi- 

10 valued mask in functional form. The DCT transform applies the transform of the sum 
of functions in the same manner as taking the transform of each individual function 
and summing the results. Furthermore there are unique spectra in each case that 
should be considered separately for a fully integrated analysis. The individual region 
masks obtained by decomposing the raw common mask for the Lena image appear in 

15 Figure 11. Notice that although there are only three images, the fourth region channel 
(the background) is implicit. 

The 2D fast DCT algorithm is used to transform each binary data set into the 
frequency domain for spectral analysis. The spectra for the upper three regions 
appear in Figure 12. Please note that a log scaling of the coefficients is used in the 
20 images to exaggerate the appearance for display purposes. Notice how the low energy 
content is dispersed through the spectrum in each case. Even at such low energy 
levels, the energy is localized in some areas. 

It is apparent from the spectral results of Figure 12 that the magnitude of the DCT 
coefficients decreases rapidly in moving down each spectrum. The interaction of the 
25 content retained for each partition is used to guide the classification procedure. The 
interrelationships that exist in each binary spectrum must be investigated further. 

A plot of the overall summed spectra in short integer format appears in Figure 13. 
The conventional zigzag ordering is used to re-group the coefficients before 
quantization. Note thai only the first half of the data in reflected in the plot The 
30 outer 8K are omitted since they do not change much from the DC level. 
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The first consideration observed in the plot is that the coefficient magnitude 
deteriorates at an alarming rate over a relatively small number of coefficients in the 
first portion of the spectrum. The second observation is that most of the spectral 
energy is concentrated in the first 12%. of the data. Together these observations 
5 suggest that implementing a low pass spectral filtering stage, followed by careful 
quantization will reduce the amount of data that must be retained to capture the region 
formation concepts at the lowest level ( in this common mask partitioning scheme). 
Note the apparently large dc offset. Part of this level is introduced by scaling the 
original wavelet data sets to a unified range. The rest of the offset is due to the DCT 
10 transform. 

Low Pass Filtering the Common Mask Spectrum 

The total mask spectrum must be filtered carefully to reduce the excess content that 
has little contribution to the region trends that exist in the retained coefficients. The 
main concern is to determine how to quantify the filter size such that a dynamic 
15 implementation may be obtained that will work accommodate any arbitrary image 
sizes. 

The first step taken towards finding a dynamic solution to this problem comes from 
experimentation. In looking at the result of using zigzag ordering, coefficient 
magnitude and significance deteriorate at some exponential rate. Experimental 
20 evidence suggests that as the size of the original image increases, the number of 
coefficients that must be retained to construct a similar quality mask increase on a 
logarithmic scale. This is an important observation since doubling the original image 
dimensions translates into a log 2 increase in the number of retained spectral 
coefficients. 

25 Assume that p is a general log base to begin our discussion. If the filter size for an N 
by N common mask spectrum has been determined experimentally as 4>n. then what is 
the size of the filter for a 2N by 2N mask spectrum? If the unknown filter size as (fen, 
then the following simple logarithmic relationship can be used to compute the new 
filter size. 

128 

— - ^SpOMM 

(Eq.III.8) 

1/2 

4>2N = $N " P 
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A number of experiments were conducted to determine a reasonable filter size that 
5 can be used as a base for filtering common mask spectra. It has been determined that 
for masks of size 128 by 128, a filter size of 40 to 45 diagonal rows yields reasonable 
results. The table in Figure 14 give the filter sizes for raw mask sizes using 1.375 for 
p. The logarithmic base p can be used for final calibration in the current design 
implementation. 

10 Notice the filter size specified for a common mask spectrum of 32 by 32 is given as 
32 diagonal rows. This value is selected to calibrate the filter sizes for the larger data 
sets. The results indicate the filter size required for a first cut of the common mask 
coefficients does not have to be exact since the distribution of the spectral content 
may change slightly from one image to the next. However, it must be large enough to 

15 capture the majority of the most important spectral content for each region partition. 

Spectral Quantization Techniques for Filtered Coefficients 

DAC has developed two techniques for quantizing the filtered mask spectrum. The 
first approach is basically a modified uniform quantization procedure that is used to 
20 verify the concepts. It proved useful in confirming the algorithms and developing the 
concepts, but it lacks robustness and flexibility. The second technique is a general 
approach that is much more robust and elegant. It is dynamic since it tracks the 
magnitude of the spectral content in determining how to quantize the coefficients. 

A First Quantization Approach 

25 The first approach combines the filter dimension determination into the quantization 
procedure. A base mask size of 128 by 128 is use as the starting point in this 
approach. Assuming that a logarithmic relationship exists for coefficient importance 
along the diagonal order, a quantization technique can be developed to exploit the 
relationship. 
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The main premise is that there are bands of spectral coefficients that can be used to 
guide the quantization procedure. The importance of the coefficients in each band 
decreases along the spectrum. This concept is illustrated in Figure 15. 

The number of diagonal zigzag rows to include in each band is a function of the filler 
5 size and the original common mask size as well as the spectral content. As a first 
approximation, a table look up technique is used to determine the number of rows to 
use in each band. The base band sizes determined experimentally to give reasonable 
partition and reconstruction results for a 128 by 128 mask are given in the table in 
Figure 16. 

10 

Notice that the number of diagonal rows doubles in each step. The number of bits 
selected to quantize each band decreases from one band to the next in the following 
fashion. 

• Band 1:16 bits 
15 • Band 2: 10 bits 

• Band 3: 8 bits 

• Band 4: 8 bits 

A series of band sizes were found experimentally of mask dimensions by powers of 2 
increments. The results are in the table of Figure 17. These values are used to 
20 partition the spectral content for quantization. This information together with the 
number of bits used in each band and the filter size measurement determines the 
common mask overhead of an image. 

The mask overhead based for this quantization scheme is tabulated in the table in 
Figure 18 for the corresponding image sizes. There is an additional overhead header 
25 included in this result for mask reconstruction on the decoder side. Note that the 
mask overhead is the same size for both grayscale and color images since it is 
generated from the intensity information. However, the percentage overhead in the 
color case is reduced by a factor of 3 since it can be distributed over 3 channels. 

After applying the inverse DCT on the quantized coefficients a low pass 
30 approximation of the original common mask is obtained. It is this result that is used 
to guide the region channels in classifying and processing the wavelet coefficients. 
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The result indicated in Figure 19 is obtained by applying this technique to the first 
wavelet transformed level for a 256 by 256 Lena image: From Table 17, the overhead 
for this mask is 1021 bytes. 

Notice the high energy changes have been eliminated by low pass filtering (especially 
5 the changes that occur over small portions of the image). However, the mask does 
succeed in capturing most of the essential parts of the original image. This mask is 
common to all orientation at this level and used to partition the coefficients into 
region categories. 

A Second Quantization Approach 

10 There are some apparent problems with the underlying procedures used for 
quantization in the first approach. The first of these is that the coefficient band 
classification is fairly rigid and not that flexible. The filtering and quantization stages 
are tied together and fully independent. Affecting one parameter has consequences to 
the other. It is true that a logarithmic relationship exists in the spectrum that can be 

15 exploited to determine a reasonable filter size, but the quantization step must be 
tailored from the filtering stage. 

DAC is currently implementing another quantization scheme for mask filtering that is 
much more dynamic and flexible. Quantization is no longer tied to the filtering stage 
and the coefficient banding technique of the previous approach. The current 
20 technique follows the actual content of the spectrum in forming quantization 
categories. In this way, coefficients are grouped into categories according to the 
energy gradient of the spectral distribution. The concept is illustrated in Figure 20. 

The rate of spectral decent together with position information within the spectrum can 
be used to quantize the number of bits used to categorize the data. This process can 
25 be controlled precisely. The procedure can be calibrated as required by studying the 
effects of changing image dimension in terms of an optimal quantization change. 

The previous quantization technique can be used to estimate the mask overhead for 
30 the new scheme proposed here. Consider the same base conditions presented for the 
banding approach applied to the 256 by 256 Lena image. The spectral filter size in 
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this case is 44 diagonal rows. If 4 quantization intervals are used at 16, 12, 8 and 4 
bits each the mask overhead can be estimated. Let N ; be the number of coefficients in 
each interval, bj the number of bits used to quantize this interval, and 0 M be the total 
mask overhead. A rough estimate of the new packed size is calculated in Eq.III.9. 

3 

10 iB ° (Eq.III.9) 

O m = 6x 16 + 39x 12+ 186 x 8 + 804 x4 

0 M = 5268 bits (- 659 bytes) 

15 

Even though the actual overhead has not yet been determined, from the insight gained 
in using the first quantization approach a saving of 30% is not unreasonable^ This 
promises to be a substantial saving in the mask overhead. There are ways to utilize 
20 saved overhead. The first is to reduce the amount of mask information in the code 
stream. This approach will retain the same mask quality leaving extra space for 
refinement information in the code stream. The second way to utilize the saved 
overhead is to retain a better quality mask in keeping the overhead size constant. 

The Common Mask and Multi-resolution Classification 

25 A translation technique is required to apply the region mask at other resolution levels. 
The simplest approach is to look at the significance of the mask coefficients in a small 
block to determine an appropriate value for the next resolution level. This process is 
repeated until a mask for each resolution is obtained. The most logical approach is to 
take the largest coefficient through to the next level. 

Mvtuijij = MaxOMv^ , Mv^^ , Mv^^, , Mv^^,) (Eq.ID.10) 

35 

Some heuristic approaches have been investigated for resolution translation. One 
approach is to make decisions based on the sum of the individual mask coefficients. 
The upper level masks are affected by either enlarging or shrinking the encompassed 
40 coverage regions. However, there is currently no conclusive evidence confirming the 
validity of this approach and more experimentation is required. 
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It is worth meniioning at this point, that DAC has begun investigating the wavelet 
kemei ROI mask formation technique of verification model (VM) 3.0 as a method of 
resolution translation for other transformation levels. Preliminary investigations 
indicate this technique introduces a gradient bias for regions of higher classification 
5 level in the translation. A combined hybrid approach may have some advantages. 

The rest of the common mask hierarchy obtained by using the simple A to 1 resolution 
translation techniques and observing the most important region as key is given in 
Figure 21. The complete set of region masks is used to classify regions of importance 
in the wavelet transform domain according to coefficient magnitude. 

10 Misclassification and the Common Region Mask 

It should be noted that while it is true that misclassification occurs in this technique, 
there is a tradeoff between code stream overhead and region channel accuracy. The 
original raw common mask size is 32 kBits for a A region mask. The basic uniform 
quantization scheme compresses this amount to 8 kBits. The modified scheme 
15 suggests that mask compression of 5 or 6 to 1 can be achieved with a tradeoff between 
mask accuracy and code stream overhead. 

The low pass filtering and spectral quantization steps can be tailored to deliver similar 
quality masks for most images in a dynamic implementation. In addition the partition 
can be studied in regards to how the original grouping is affected by the filtering step. 
20 For the Lena image example the original partition is 10/15/25/50%. After filtering the 
coverage ratios used for bit budget calculation and the code stream distribution (see 
multiplexer in Ch.V.) changed to approximately 9/31/33/27%. There may be some 
advantages in adjusting the original partition such that the data coverage ratios are 
controllable in the final stages. 

25 The question of miscoverage must be addressed in all lossy region formation 
techniques. It is the subsequent modules that are affected by the misclassified 
information. Thus the interaction between the DCT region formation module and the 
subsequent sorting and packing modules must be understood in any attempt to obtain 
the best region channel results for this classification scheme. 

30 
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Subsequent stages in the compression algorithm are already tailored to process 
information in a descending level of importance. Most wavelet implicit quantization 
procedures process the coefficients in bit plane order of importance. What this means 
5 in terms of any region formation scheme is that the information encompassed in each 
region is further partitioned by bit level processing. In the common mask approach, 
further processing causes the most important misclassified coefficients to filter to the 
top at a faster rate than misclassified coefficients of lower importance. Effectively bit 
level processing can be considered as a second partitioning stage that deals with 
10 misclassification. 

There is a tradeoff between the accuracy of the original region partition and the 
additional bit level processing overhead placed on subsequent sorting stages. It is the 
final processing stages that implicitly accomplish the final coefficient partitioning. 
DAC is currently using two bit plane partitioning core technologies for both grayscale 
15 and color images, and binary partitioning for bi-level images is now under 
implementation. 

Bit Plane Sorting for Common Mask Miscoverage 

The concept of bit plane sorting is a well know technique used to organize priority in 
a distributed list of data. ID sorting a list of information generates a map specific to 

20 that list in an optimal fashion. The largest values are mapped first as the list 
transposition routine progresses. The order of decent is the standard order used to 
classify information if considered from the normal processing standpoint where no 
regions are involved at all. As a result, most of the misclassified information is 
considered for further classification and are thus biased to be included first in the final 

25 code stream order. The only difference is that now there is a region description 
overhead as well as the final mapping overhead of sorting the region of interest 
partitions in a hierarchical fashion. 

Currently DAC has not documented the exact overhead introduced by region 
processing in comparison to normal processing modes that do not use regions. But 
30 the final results obtained by using the common mask approach are encouraging to say 
the least. At this level the underlying routines do not care where the information to be 
sorted originated. The ordering technique will run to completion whether region 
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channels are involved or not. The original misclassified coefficients generated in the 
first cut will filter through to the final ordering set by the sorting stage. There is an 
important tradeoff here between the sorting cost and the region overhead that must be 
considered in the final design implementation. 

5 Bit Plane Ordering for Common Mask Miscoverage 

DAC has developed a quad tree ordering technique {see EQW Ch.4) to track 
coefficient importance in both normal and region based processing modes of 
operation. The normal quad tree approach is to track the leading one position through 
each block under consideration in a recursive fashion. A map is generated in the 
10 process for each entry based on importance such that decoding follows the same 
order. At this basic level the quad tree order is an efficient method of processing the 
multi-resolution decomposition image information in an effective manner. 

The tracking or positioning overhead introduced by mapping the data is generally 
smaller than in the ID sorting case. Retaining vertical and horizontal positioning 

15 information is highly advantageous (in most image processing applications) since the 
underlying technology is generally consistent in this medium. The ID sorting 
approach sacrifices the natural vertical correlation that may exit (although it can be 
partially recovered by bit level entropy coding in the sorting routine.) However each 
of the two sorting techniques can be used from an operational standpoint to produce a 

20 similar net effect. One may produce slightly better results than the next, but the 
control architecture is consistent. Each final ordering routine can be tuned to meet 
specific processing needs. The ID sorting approach may be a useful candidate for bi- 
level image processing, and DAC is currently investigating this possibility. 

The quad tree approach has currently been modified to operate in arbitrary region 
25 processing modes. The ID sorting routine tracks the importance in a list of 
information specific to the block under consideration. That is the region partition is 
distributed in ID hierarchical lists of information. The original preordering of the 
coefficients is captured in the common mask procedure. Thus the decoder map 
already exists for the inverse ordering. 

30 Initially DAC's region processing quad tree begins by building an information map 
for each region channel to guide the classification. The core engine has been 
modified to drops blocks not contributing to the final ordering in each region 
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category. Currently the overhead comparisons for each sorting case have not been 
documented. The results will be available from DAC once the region based quad tree 
design implementation is finalized (currently under implementation at DAC.) 

In each case, the misclassified coefficients obtained from the original preorder mask 
5 partition are considered again for ordering in the sorting stages that follow. 
Effectively, the most important misclassified coefficients still have the opportunity to 
make important contributions to the final code stream as well as improving the 
reconstruction quality measurements in the decoded image. The difference between 
this approach and most efficient classification approaches is that the original 

10 processing order used of the efficient scheme is somewhat distributed through each 
region channels. However the misclassification in each region channel is forgiven to 
a certain extend the subsequent bit level sorting module. One of the prices that must 
be paid for any automatic region classification scheme is that the optima! processing 
orders that exist in standard approaches must be considered in a different context. 

15 The original ordering concepts used for standard procedures must be extended to 
include other ordering techniques specifically tailored for operation in arbitrary region 
processing. 

General Region Processing Concerns 

There is one important hurdle that must be overcome in the design of any arbitrary 
20 region processing technology. There are times when rigid tile or block control is the 
optimal way to process image information. However, it is difficult to achieve 
arbitrary accuracy in a strict processing order. DAC's ID and 2D sorting techniques 
address this issue in design implementation. In our opinion arbitrary region 
processing channels can be defined in a still image environment. Careful study and 
25 analysis suggests that this is a viable approach not only for the DCT common mask 
approach but also for other arbitrary region formations techniques that have not yet 
been developed. Furthermore region classification may have interesting properties 
that can transferred to video processing environment where the region of interest 
concept and the object motion vectors can be considered together in a unified 
30 approach to region of interest processing. 

DAC's region technology in its original form was designed to include region 
processing capabilities from the start. All internal modules can be operated in both 
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normal and region based modes of operation. The MUX architecture {introduced in 
Chapter V) is designed to meet operational requirements in both normal and region 
processing modes. It is possible to extend the notion of block based image processing 
to a level where a block of information is interpreted in an arbitrary way that depends 
5 only on the underlying core processing technology. In this way, blocks can be 
considered as individual processing units each fully capable of making a contribution 
to the final code stream. Furthermore control architecture can be developed to fully 
exploit the region processing capability. All region formation schemes, whether 
arbitrarily defined user regions of interest, automatic region formation schemes or any 
0 other specially designed region classification technique, can be processed in a 
common framework. The same overall processing architecture can be tailored to 
operate in an optimal fashion for both normal and region processing modes of 
operation. 

Additional Processing Concerns 

One of the important concerns in the JPEG-2000 community regarding the 
development of a new compression standard is that it must be able to support ROI 
processing. There are a number of techniques that have been tested with some 
success. Some of these techniques are listed below. 

• Sequence based mode. When this mode of operation is used, each set of data 
encapsulated by a specific region is treated as a separate sequence. The sequences 
are coded independently. 

• Scaling based mode. In this mode of operation, the magnitude of each ROI 
coefficient is increased using a predetermined shift factor. If multiple ROIs are 
used with increasing levels of importance, multiple shift factors distinguish each 
ROI. The effect of this scaling technique is to force the ROI coefficients to be 
encoded first. 

• ROI from the middle. In this mode of operation, a concern of coding an ROI in 
the middle of the bit stream is addressed. This is useful for progressive 
transmission of images where the user is given the opportunity to focus on a 
specific ROI before the complete set of image data arrives at the decoder. This is 
important for medical image processing. 
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• ROI without sending mask information. This mode of operation is a special case 
for scaling based mode. The coefficients for each region are scaled such that the 
magnitude of each ROI exceeds that of the ROI of less importance. The decoder 
knows which ROI the coefficients belong to based on the magnitude. 

5 The ROI coding technique that DAC has developed generally falls into the sequence 
based category. But internal modules have been tailored to operate in both block 
based and arbitrary block sized processing units. DAC is currently studying the 
different mask and region formation techniques to determine the effect each ordering 
and partitioning approach has on the overall organizational structure required for 
10 region channel processing. The current implementation includes a scalable ROI mode 
of operation implemented without actually shifting the ROI coefficients encompassed 
by the classification. Data scaling is handled in the MUX control architecture. 
Internally, the scaled version is a special case that can be implemented in MUX 
control. 

15 Preliminary study has begun for the last of the four categories cited above. ROI from 
the middle implementations can be addressed by using resolution a progressive 
processing order. This type of ordering is supported in the MUX architecture. DAC 
is currently considering a design implementation for client side region of interest 
requests in a resolution progressive architecture for client-server region interaction. 

20 ROI without mask information makes sense in theory. However it places a huge tax 
on the subsequent sorting and classification stages. Generally speaking, the overhead 
cost introduced by additional sorting is comparable to the size of a bitmap mask that 
could have been used to classify the data initially. The shift introduced to each ROI 
must be processed in the sorting stage. Eventually sorting must traverse from top to 

25 bottom. Coefficient scaling under tight control has some definite benefits in many 
cases. However it would be highly inefficient in a general region processing 
architecture because of the high sorting costs as a result of traversing all region 
channels. In a region formation scheme of 4 or more regions, the total bit level 
difference between all ROI is too large to process entirely. On the other hand, if the 

30 resolution of the overall partition is reduced (less bit levels), less room is available to 
form an accurate region classification. There is some common ground that must be 
explored in order to determine the optimal processing mix. 
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Intra-Region Coding 

The segregation of image primitives into regions provides a basis for accessing the 
image contents. Compact representation of image primitives is achieved via intra- 
5 region coding. 

Independent coding units (ICUs) are the building blocks for intra-region coding. In 
RICS, intra-region coding with ICUs is designed with a set of objectives. 

(I) Full data independence. From the notion of ICU, the encoding and decoding of a 
region is done without referring to the data of any other regions. 

10 (2) Allowance for multiple coding schemes. Each ICU may be coded using different 
coding schemes. New coding schemes can be added to the system (modular 
openness.) 

(3) High coding efficiency. Depending on the characteristics of the primitives in a 
given ICU, one or more efficient coding schemes can be selected to produce a 

15 code block with high compactness. 

(4) JPEG and JBIG transcodability. It is preferred to have a single JPEG 2000 
coding platform that also accommodates JPEG and JBIG modes of coding. 

(5) Scalability of code stream. 

(6) Error resilience. The ICUs are natural blocks for bit error localization. 

20 (7) Parallelism. There is no data dependence between ICUs and so the encoding and 
decoding of all ICUs can be performed in parallel. 

Intra-region coding in RICS involves the following steps. 

(1) Choosing a coding scheme. 

(2) Determining ICU structure. 
25 (3) Coding. 

(4) Pre-packing. 

In the rest of this section the first three steps are described. 
Choosing a Coding Scheme 
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For coding an ICU, lhe current RICS design employs six categories of coding 
schemes. 

• Zigzag DCT Coefficients Packing (JPEG baseline scheme) 

• Embedded Quad tree (or similar schemes) 
5 • Embedded Zero tree (or similar schemes) 

• ID Progressive Sorting Schemes 

• Block Based Quantization 

• JBIG Coding Algorithms 

There is no general restriction on what schemes have to be used for coding a given 
10 ICU. However, certain preferred embodiments may apply. For example, if DCT is 
used at the transform step, then the zigzag DCT coefficient packing might be 
preferred (but not restricted to). 

Choosing ICU Structures 

While being a logic concept, an ICU still has a geometric structure. The ICU 
15 structure is directly relevant to the coding scheme chosen. Currently, the RICS 
recognizes three types of ICU structures (Figure 22). 

Type 1 ICU Structure: 8x8 Blocks 

This type of ICU is designed exclusively for use in JPEG mode, although it is not 
necessary that DCT transform coefficients be coded this way. 

20 If there is no region defined in the image and the JPEG mode is chosen, the entire 
image is paved with type-1 ICUs. 

If a region R is to be coded using JPEG mode, the set of ICUs that covers R form a 
unique, minimal pavement for/?, in the following procedure. 

Step L Let y ™ be the first row from the top that has at least one pixel in R, y m» the 
25 last row, jc™ the first column from the left, and xm*x the last column. 

Step 2. Let (jc ™,y «,«.) and (x^y™*) be respectively the top-left and the 

bottom-right corners of the bounding rectangle of R. 

Step 3. Starting from the top left comer, pave the bounding box completely with 
type-1 ICUs. 
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Step 4, Remove those ICUs that cover no pixels in R. 

After this procedure, the remaining ICUs form the minimal pavement of R (Figure 
22). 

Type 2 ICU Structure: Rectangles 

5 Type-2 ICUs are rectangles. Except for the JPEG and embedded zero tree schemes, 
any other intra-region coding scheme uses type-2 ICUs to pave and code a region. 
There is no general restriction on the dimensions of a type-2 ICU. It is usually 
defined by the selected coding scheme. For example, both embedded quad tree and 
explicit block based quantization can use arbitrary sized rectangles, with some 
10 preferred embodiments (such as the 64x64 blocks used in EBCOT of VM 3.0 (A)). It 
should be noted that once a region is paved by type-2 ICUs, different coding schemes 
can be used for each ICU. 

For subband decompositions, no type-2 ICU is allowed to cross subband borders. 
Essentially, this means that type-2 ICUs support only various types of intra-band 
15 coding methods. 

Type-3 ICU Structure: Pyramids 

The third type of ICU is the pyramid structure, which is used by various inter-band 
coding methods, such as the embedded zero tree. In the current RICS design, there is 
one limitation on the region definition procedure for type-3 ICUs: a region must be 
20 specified in the top-down manner, from lower resolution to higher resolution in the 
decomposition pyramid. That is, for a given region definition R in LL, every element 
in R defines a set of type-3 ICUs (i.e. in three spatial orientations, respectively). 

Let C u {iJ) be a wavelet transform coefficient at spatial location in the LL 

subband, C; (m,«) be a wavelet coefficient at spatial location (m,/i) in the subband 
25 at level / (1=1,2,...) in orientation r (r=],2,3). Assuming that a 5-level wavelet - 
decomposition is used, the three type-3 ICUs defined by C u (/,/) can be represented 
in the following fashion. 

In orientation 1 : 

JCJM) U {Ci(2i , ,2y),Ci(2i" + l,2ACi(2|- i 2y + l) i Ci(2i+I,2y + l)} u ... 
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In orientation 2: 

IClUJ)) U [C 2 4 (2i.2j),Cl(2i + 1.2j), C« 2 (2/.2y+l), C, z (2i +1.2/ + 1)} U ... 
In orientation 3: 

[C\{iJ)) U {C< 3 (2/,2/), Cj(2*+l,2y), C, 3 (2i\2/ + 1), C 4 3 <2/ +l,2y +1)} U ... 

5 Notice that the primitive C^HJ) is not included in any the three ICUs. Instead, the 
LL subband is coded differently, using the type-2 ICUs. 

Coding Type-l ICUs 

Type-l ICUs are defined exclusively for coding 8x8 DCT coefficient blocks. Coding 
of type-l ICUs follows the JPEG baseline algorithms. 

io Coding Type-2 ICUs 

Once a region is decomposed into a set of non-overlapping type-2 ICUs, each ICU is 
coded independently, and the collection of codes for all ICUs is the coded 
representation of that region. In this subsection we describe three techniques that may 
be used for coding type-2 ICUs. 

15 Using Embedded Quad tree 

Embedded Quad tree Wavelet (EQW), Figure 24 is an efficient and fast method for 
type-2 ICU coding. This technique implements an embedded progressive sorting 
scheme in a quad tree-like structure. In contrast to inter-band coding methods, the 
EQW explores the intra-band self-similarity of the wavelet decomposition 
20 coefficients. The EQW-produced code-stream realizes scalability by pixel-precision 
(the scalability by spatial resolution is realized by the multiplexer.) 

Depth-First and Breadth-First Quad trees 

The EQW method can be used for both lossless and lossy coding. In lossless coding, 
the primitives to be encoded in the ICUs are coefficients of a reversible wavelet 
25 transform. In lossy coding, after the wavelet transform, each transform coefficient is 
represented in a fixed-point binary format - typically with less than 16 bits - and is 
treated as an integer. 
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In each ICU, the maximum coefficient magnitude M is determined. Then a value N is 
found which satisfies the condition 2 N <M < 2 N * X . The EQW works in a bit-plane 
manner: the initial bit-plane is set to 2", followed by 2*~\2 N ~ 2 ... and so on. A 
binary significance map is produced for every bit-plane by comparing coefficients in 

5 power of 2 increments. 

Figure JV.3 illustrates the EQW encoding process on a single bit-plane. Two working 
lists are used. One is called the list of significant primitives (LSP). Each entry in 
LSP corresponds to an individual primitive in the ICU. The LSP is initialized as an 
empty list. Another list is called the list of insignificant blocks (LIB). Each entry in 
10 the LIB is registered with the coordinates of the top-left corner of the block, together 
with the width and height information. Each LIB entry represents an individual 
primitive of width and height equal to 1. At the highest bit-plane 2* the ICU to be 
encoded is put into the LIB. A temporary list of insignificant blocks (TLIB) is used 
for refreshing the LIB after each bit-plane coding pass is completed. 

15 There are two methods used to add the four sub-blocks into the LIB that occur as a 
result of the quad tree decomposition. The first method, known as depth-first quad 
tree coding, is to insert the four sub-blocks into LIB at the position of their parent 
block. In this case, the four child blocks are evaluated immediately after the parent 
block. This rule is applied recursively until no more subdivision is possible. This 
20 means that control returns to the next entry in the LIB only after the present entry is 
completely encoded up to the highest resolution level. The second method, known as 
breadth-first quad tree coding, is to add the four sub-blocks under consideration to the 
end of LIB. In using the breadth-first process, all parent blocks at the same level will 
be processed first, followed by their respective children blocks. 

25 After all entries in the LIB have been processed, the entries in TLIB can be reordered 
according to certain pre-defined set of rules (this is an optional step.) Experimental 
evidence suggests that a higher PSNR can be achieved by using an effective 
reordering scheme. Then the LIB is replaced with the TLIB and the coding resumes 
at the next bit-plane. 

30 The next step is the refinement pass. The N th bit is output for entries in the LSP. 
Then, the process resumes using .the new LIB and a new bit-plane set to Error! 
Objects cannot be created from editing field codes.. 
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EQW with VLC 

In order lo use variable length coding (VLC), the standard EQW algorithm of the 
previous section is modified as follows. 

Step 1. Initialization: output n satisfying the inequality 2" < max{|C (> |} < 2"* 1 , set the 
5 list of significant primitives (LSP) as an empty list. Put the ICU to be coded into the 
LIB. 

Step 2. Bit-Plane Sorting: for each entry of the LIB, perform quad tree coding. 

If all primitives within the block are insignificant, output a 0 bit and add this block to 

the temporary LIB (TLIB). 

10 Otherwise, look at the significance of the four sub-blocks. Then according to the 
VLC table, output the corresponding VLC sequence (see the table in Figure 25). If 
significant, sub-blocks that are not a single primitive are inserted into the LIB at their 
parent position. If insignificant, they are added to the TLIB. If the sub-blocks are 
single primitives, they are added to the LSP. Sign bits are output only if units are 

15 significant. If insignificant, the units are added to the TLIB. 

Step 3. Reordering (optional). 

Step 4. Refinement: for each entry in the LSP, except those included in the last 
sorting pass, (i.e. with same/i), output the /ith most significant bit of |C tf |. 

Step 5. Quantization Update: decrement n by 1 and go to step 2. 

20 Some other researchers have also proposed the embedded coding methods where quad 
tree is used to explore the intra-band similarity [m . However, some major 
differences exist in how the quad tree operates. The work of [2] has suggested that 
empirically the quad tree decomposition should stop at size 16x16. However (as a 
result of our experience and experimentation) this size may not always be the optimal 

25 choice. Even without VLC, the efficiency of DAC's EQW is comparable with EZW. 
In fact, for a 2x2 block, if it is insignificant, one zero can represent four zeros. 
However, when a 2x2 block is significant, quad tree coding is inefficient. 



F. Kossentini et al. "Embedded Quadtree Predictive Codec", JSO/IEC/JTC1/SC29/WG1/N667. 
12J Taubman, D. el al. "EBCOT: Embedded Block coding with optimized truncation". ISO/IEC/ 
JTC1/SC29/ WG1/ N1020R. 
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Using Block-Based Quantization 

The embedded quad tree (EQW, EQPC, etc.) is a special case of the more general 
block-based quantization techniques. Other block-based quantization algorithms, 
. such as EBCOT, can also be very efficient for type-2 ICU coding. 

5 

Using JBIG Algorithms 

It is noted that in all implicit quantization schemes using progressive bit-plane coding 
methods, including the embedded zero tree and the embedded quad tree approaches, 
10 at each bit-plane, the ICU coder is actually dealing with a bi-level image (the 
significance map.) Therefore, by adopting those efficient JBIG algorithms as ICU 
coders in a JPEG 2000 system, one can realize two modes of interplay with JBIG. In 
the following EQW is used as an example to illustrate this idea. 

15 In the first mode, the transcode mode, the JBIG ICU coder and EQW ICU coder 
provide a two-way transcode between JPEG 2000 and JBIG (Figure 26). In this 
transcode, the EQW procedure is called for bi-level type-2 ICUs, with the total 
number of bit-planes being set to one. Note that the sign bit coding in the EQW 
algorithm should be skipped. 

20 In the second mode, the embedding mode (Figure 27), the proper JBIG routines are 
called as the bit-plane coder in the EQW routine for certain bit-planes and for certain 
decomposition sizes (the quad tree decomposition stops at this particular size and the 
coding is handed over to the JBIG routine). Since the algorithms for bi-level data 
coding are also evolving with the compression technology, having an embedding 
-25 JBIG mode reserved in the JPEG 2000 system can make the Standard evolve with its 
sister technologies. 

Coding Type-3 ICUs 

Type-3 ICUs are defined to support various inter-band coding methods for wavelet 
transform coefficients. In particular, the methods of EZW, SPIHT, etc., are known to 
30 be efficient approaches for coding the type-3 ICUs. In testing these approaches, it is 
noted that the standard versions must be modified slightly to make them fit into the 
RICS architecture. Normally, these algorithms implement the ICU coding and 
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multiplexing steps into an integrated procedure. A simple modification is required to 
bridge the existing architecture to the R1CS architecture. The two steps must be 
separated, which is straightforward. Since the multiplexing stage in R1CS can 
effectively simulate the performance of zerotree-based schemes (refer to Ch.V), this 

5 particular split in functionality has virtually no impact on the attainable coding 
efficiency for standard schemes. Instead it adds more flexibility and openness to the 
existing architecture. 

Intra-Region Coding with Different Types of ICUs 

Generally there is a need for using different types of ICUs for a intra-region coding. 

10 For example, in coding the wavelet transform coefficients, the LL subband is usually 
encoded differently than the other high-pass subbands. Because different sets of 
primitives may possess different statistical characteristics, providing several coding 
scheme choices in the intra-region coding module may offer a higher coding 
efficiency. In practice, the following combinations are useful. 

15 • In a wavelet decomposition, coding the LL subband with type-2 ICUs whereas the 
other subbands with either iype-3 or type-2 ICUs. 

• Tiling an image into several regions, with some of the regions paved with type-1 
ICUs and others with type-2 ICUs. 

• Using different coding schemes in different type-2 ICUs. 

20 Intra-Region Coding without Any ICUs 

Intra-region coding can be performed without specifying any ICUs in a particular 
region. In this case, the entire region is considered an ICU. If the natural geometric 
shape of the region fits into any of the three ICU categories, the appropriate ICU 
coder can be used. If the region has a very irregular shape, then the ID coding 
25 method can be an efficient approach. Once the data for a particular region is scanned 
in a certain pre-defined order to form a ID stream, the following ID progressive 
sorting algorithms can be used to produce an embedded code-stream. 

ID Progressive Sorting Algorithms 

The EQW algorithm can be readily extended to ID cases where a binary partition is 
30 used instead of a quad tree decomposition: 
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Let L~ {c,-} be the ID list to be encoded, LSP the list of significant primitives, LIS 
the list of insignificant subsets, TLIS the temporary list of insignificant subsets. 

Step 1. Initialization: Output n satisfying the inequality 2" < max{|c. |) < 2"* 1 , set the 
LSP as an empty list. Put the set L into LIS. 

5 Step 2. Bit-Plane Sorting: For each entry in the LIS, perform the binary partition. 
If all primitives within the subset are insignificant, output a 0 bit and add this subset 
to the TLIS. 

Otherwise, output the two bits that reflect the significance map. For those subsets, if 
they are not single primitives, insert them into the LIS at their parent position if they 
10 are significant, or add them to the TLIS if they are not. If the subsets are single 
primitives, add them to the LSP and output the sign bit if they are significant, or add 
them to the TLIS if they are insignificant. 

Step 3. Reordering (optional). 

Step A. Refinement: for each entry in the LSP, except those included in the last 
15 sorting pass, (i.e. with same n), output the nth most significant bit of |c,-| ; 

Step 5. Quantization Update: decrement n by 1 and go to step 2. 
Processing with a Multiplexer Architecture 

The JPEG-2000 committee has been investigating many emerging compression 
20 technologies in order to define a new still image compression standard. The emerging 
standard will address both present and near future image compression needs. The task 
of selecting what technologies to include in the new standard is not easy given the rate 
at which technological advances occur in this area. The primary focus of DAC has 
been to develop a still image compression engine that addresses the issues set forth by 
25 the standards committee. 

The concept of using a multiplexer (MUX) has been adopted in some standardized 
processes of information coding such as MPEG. In contrast, incorporating a MUX as 
an integral part of a still image compression engine is rather unique, and there has 
been very little research conducted in this area. However it will be shown that the 
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MUX concept is extremely useful in the development of a general purpose still image 
compression engine. 

Background 

One of the concerns in developing a new Standard is the layout design of the encoded 
5 bit stream. A protocol must be developed to guide the new standardized compression 
procedures. Some of the important considerations in this area are listed below. 

• A well defined bit stream. All syntactic and semantic aspects of the encoded 
image format must be defined. The JPEG-2000 standard will encompass a wide 
variety of compression needs. One to one relationships will exist between fields 

10 that exist in the bit stream and functionality that exists in the core compression / 

decompression engine. The number of fields that exist in the bit stream is directly 
related to the overall functionality encompassed by the new standard. 

• A highly degree of data accessibility. The bit stream must be organized in such a 
manner that it need not be decoded in its entirety to interpret the physical meaning 

15 of the compressed data. Highly accessible data requires that the encoded data is 

divided into logical units each corresponding to a specific part of the original 
uncompressed information. The way this was accomplished in the existing JPEG 
standard was to partition the original image into 8x8 blocks. Each block was 
compressed using the DCT. Quantization tables were developed for coding the 

20 coefficients such that the reconstructed image quality was degraded in a standard 

fashion. However, strict block based data accessibility may not be a suitable 
medium for many of the current image processing needs. 

• Dynamic re-orderability. Dynamic re-orderability is a data access issue relating 
to the degree thai encoded information can be reorganized to suite user specific 

25 needs for an image under consideration. This type of data access is an important 

concern for many applications, especially in the medical field. 

• Built in error resilient In any transmission medium there is always the 
possibility of incurring bit errors. The encoded bit stream must have a certain 
degree of error resilience to address this concern. 

30 Using a MUX to organize the encoded bit stream can address these issues. The first 
step is to divide the encoded information into logical groups. Organizing the data in 
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ICUs leads naturally into a high degree of accessibility. Each ICU is an independent 
unit. The size and position information is self-contained within each ICU. One of the 
effects of using a muliplexed bit stream design is that it is inherently error resilient. If 
any logical unit incurs an error, only that unit needs to be recovered. If the encoded 
5 data is divided into an array of logical units in preparation for MUX encoding, 
dynamic re-ordering is much easier to achieve. 

MUX Design Overview 

There are five important considerations to address in the design of the MUX for still 
image compression systems. 

10 • Defining a working model. 

• Defining an optimal data processing order. 

• Capturing the data to be compressed in a suitable data structure. 

• Defining a data priority to be used for compression. 

• Packing the compressed data based on data priority and processing order. 

15 A Working Model for the MUX Discussion 

In preparation for the MUX discussion, consider that a multi-resolution 
decomposition hierarchy of data has been obtained for an image to be compressed. 
Also consider that there are three channels of data. Assume that the original image 
has been transformed from RGB to YUV for example. In addition to this, assume that 
20 the U and V channels have been down sampled by a factor of 4:1. This type of 

approach is common for lossy image compression. The data sets appear in Figure 18. 

The diagram illustrates a 4 level multi-resolution decomposition for the Y-channeJ 

and 3-Ievels for each of the U/V-channels. The wavelet transform MALLAT 

decomposition is used here for convenience. 

25 This is by no means the only model that exists for the MUX design. DAC makes the 
distinction between region and non-region processing modes of operation in their 
design implementation. A hierarchy of list structures is used to control optimal bit 
budget distribution and flow of both non-region and region bit levels through MUX 
channels. The discussion presented in the next Section- begins by introducing the 

30 concepts in normal processing modes. The normal MUX modes of operation are 
generalized later on to include specialized region and mixed processing modes. 
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Generally speaking, given a wavelet multi-resolution decomposition of data, 
coefficients of a fixed magnitude contained in a lower resolution level are more 
important to the image reconstruction procedure than coefficients of a similar 
5 magnitude at a higher resolution level. This leads to a natural ordering of the 
coefficients beginning at the lowest resolution level. The natural ordering concept is 
an important consideration in the design of the MUX. 

General Color Processing Order (Lossy Case) 

Given the multi-resolution data sets of Figure V.l, a natural processing order exists 
10 for each decomposition channel. This natural order is illustrated in Figure 29. This 
particular order reflects a level priority inherent to each data set. 

Another two orders are obtained by simply interchanging orientation data sets in a 
given level. The difference lies only in the convention used for ordering the detail 
information (i.e. LH f HL, HH). The 4-1-1 down sampling relationship that exists in 
15 the color transform domain also exists in the wavelet decomposition data since there 
is one less transform level for the U/V channels. This is one of the keen design 
considerations that can be exploited in many stages of the MUX implementation and 
encoding / decoding procedures. The data is organized into list structures to cover the 
natural processing orders that exist in the data to be compressed. 

20 Level Priority Processing Orders (Lossy Case) 

The individual orders of Figure V.2 can be combined into a single natural processing 
order for the example under consideration. The new order is obtained by interleaving 
the expressions of Figure 29 using the inherent decomposition level priority of the 
data. The result is illustrated in Figure 30. 

25 Notice that in this particular order, the 4-1-1 color down, sampling relationship is 
maintained. In terms of the reconstruction process, the complete level 4 Y -channel 
information (i.e. LL4-HL4-LH4-HH4) is required together with the U and V channel 
low pass information (i.e. LL3 in each case). This is the information the decoder will 
need first to reconstruct the low pass inverse color transform image at the lowest 

30 inverse wavelet resolution level. A "level split" on the U / V data channels at the 
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lowest inverse wavelet resolution level is used to exploit the natural relationships that 
exist for both inverse transform spaces. A similar order exists for the lossless case 
where full data sets are used. 

Color Interleave Processing Order (Lossy ) 

5 Another processing order that exists for color images in the wavelet transform domain 
is obtained by interleaving the color channel detail information. The order is 
illustrated in Figure 31. This type of ordering may be suitable for certain applications 
such as those that require the data to be encoded for a progressive download. As in 
the previous case, the down sampling relationships that exist are maintained in both 
10 color / wavelet transform spaces. 

Lossless Color Processing Orders 

As in the lossy case, similar processing orders can be designed for lossless color 
image compression where full data sets exist for all color channels. In this case, 
corresponding orientation information from each wavelet channel is related in the 
15 original inverse color space. These data should be processed and grouped together in 
the wavelet transform domain and eventually the final bit stream. 

The natural processing order for lossless color image processing is illustrated in 
Figure 32. The color interleave order for lossless image processing is illustrated in 
Figure 33. Note that in each case, the full decomposition data set for each channel is 
20 maintained. The inherent relationships that exist in the color transform space are 
maintained in the wavelet transform space for ordering and reconstruction of the 
original image. 

Capturing the Data in a MUX List Structure 

There are a number of candidate algorithms under consideration by the JPEG-2000 
25 committee for optimal quantization of the wavelet coefficients. A common approach 
used to quantize the wavelet coefficients in a progressive manner is to use an optimal 
bit plane ordering technique. DAC has testing a number of bit plane ordering 
techniques in both one and two dimensional schemes. Bit plane ordering techniques 
are generally classified as using implicit quantization for the wavelet data sets. 
30 Coefficients are packed into the final bit stream based on bit level priority. The bit 
plane processing technology will be used to introduce the data structure used in the 
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MUX design. However, ihe MUX processing discussion that follows is not restricted 
to bit level processing algorithms. 

The general ordering and processing relationships are useful for introducing the MUX 
control architecture at a basic level. However, the key to the operation and many 
5 benefits of the MUX technique is in the data structure design used to encompass the 
natural ordering concepts. The first step is to define a list structure for each level, 
channel, and orientation of data that exists in the wavelet transform space. A typical 
example of this type of structure is given in Figure 34. 

A general list structure for each level data set can be utilized in many ways. A 
10 lossless "prepack'* of multi-resolution hierarchy information is useful for exploiting 
the many relationships that exist in the data taken in MUX list processing context. 
The data and information contained in each list is used to control how the final bit 
stream is organized. All data packed into the final bit stream must conform to this 
structure. New fields may be added, but the basic operation of the structure remains 
15 the same. Most implicit quantization techniques are implicitly decodable. In other 
words, minimal header information is required for each list. The MUX architecture 
organizes the lists or units of information into a bit stream that is scaleable in terms of 
bit precision and resolution and is controllable by the many MUX modes of operation. 

As an example, suppose that EQW (DAC's current two dimensional bit level 
20 processing design) is used to capture the leading ones and refinement bits at each bit 
level in each data set. Following the order given in Figure V.3, a multi-dimensional 
hierarchy of list structures is defined for use in the ensuing data processing stages. 
Within each list (as EQW progresses), the fields in the corresponding list structure are 
updated. The bit stream at each bit level is appended to the MUX "prepack" buffer as 
25 well as its corresponding size being added to the total in the bit packing information 
field. The bit level position where processing begins is put in the high bit information 
field. The total packed list size is placed in the total packed field. Finally if more 
than one internal processing scheme is used (e.g. NULL transform mode for bi-level 
information or other internal modes of operation), the scheme used for the list is 
30 placed in the scheme field. 
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Each MUX list is fully independent, implicitly contains a full set of statistical 
information available for determining the amount of data to pack for each list and is 
optimal for organizing the final compressed bit stream. 



5 Data Priority and MUX Control 

The information contained in each list structure is used to organize the final bit stream 
based on end user compression requirements. DAC has implemented numerous MUX 
modes o operation. Two common modes of operation are outlined in this Section. 

Signal to Noise Ratio (SNR) Progressive Mode 

10 This mode of MUX places priority on the data such that the final ordering optimizes 
the SNR given a user specified image compression size. For the discussion that 
follows, assume that the user has specified the desired compression ratio in some 
manner. Assume that an optimal bit budget distribution scheme is in place. Thus the 
number of bytes that has been allocated to each color channel is known (more will be 

15 said on how to determine the optimal bit budget distribution for each channel using 
the MUX later). The information fields in the MUX list structures are used to 
determine the amount of data to pack for each list such that the SNR is optimized for 
the specified bit budget. A psuedocode explanation of this technique appears in 
Figure 35. 

20 Fields that refer to the MUX list structure appear in italics while other local variables 
appear in normal typeface. Basically the idea is to loop through each list checking to 
see whether the current processing bit level is equal to the bit level of the list under 
consideration. If the two bit levels are equal, the channel bit budget is decreased by 
the amount of data available in this particular list for the bit level in question (from 

25 MUX field pliPackinglnfolcCurBitLevel], The MUX field UCurByttsCount is 
incremented by the number of bytes available. Additional processing takes care of the 
remaining bits. The remaining bits will be considered in the channel bit budget on the 
next visit to this particular list. The MUX field cCurEiilevel is decreased by one 
such that it will again be enabled when the bit level in the main loop is decreased by 

30 one. 

This procedure optimizes the SNR since it processes the largest bit levels in each list 
first. Thus information for the largest coefficients throughout individual 
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decomposition data sets is sent first according the natural processing order outlined 
earlier. Once the channel loop exits, each MUX list will contain a field indicating the 
amount of data to pack into the final bit stream in each case. 

Resolution Progressive Mode 

5 The resolution progressive mode of bit budget distribution is a simple extension of the 
psuedocode implementation of the SNR mode of the previous Section. In order to 
obtain a resolution progressive mode, the 'Waveke: Transform level' for loop is taken 
outside of the main BitBudget / BitPlane while loop such that resolution level is given 
priority. In this manner, a lower resolution image can be reconstructed in the inverse 
10 wavelet transform stage. The resolution of the ensuing image depends upon the 
number of levels required in the end user requirements. 

Data Packing Using the MUX Lists 

Composing the final bit stream is a simple matter once the bit budget distribution 
scheme has run to completion. The packing technique can follow one of the normal 

15 processing orders outlined earlier (e.g. color level priority or level priority color 
interleave) or another access technique as required by the end user. The amount of 
data to pack for each list has been determined in the bit budget distribution stage. If 
the data to be packed for a given list is greater than zero, then the total data size, 
scheme and high processing bit level are packed as header information into the final 

20 bit stream. If the data to be packed for a given list is zero (i.e. not required), then a 
smaller header is packed to indicate a zero length list. Currently a register lype 
approach is being developed to handle zero length lists. All lists that have data 
available will set a bit position in the list register so that the decoder can determine 
which lists have made a contribution. Empty list will be flagged with a 0 bit in the list 

25 register. The list register is packed in the final code-stream. This technique will save 
the size of the list header overhead of empty lists. 

Overhead Consideration of the MUX Architecture 

There is a small overhead placed the compressed data by the MUX implementation. 
There are three header fields to pack for each MUX list. 

30 • The total bytes packed. 

• The scheme used to process the list. 
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• The highest bit plane for data in list. 

Note that if there is only one internal scheme used to process the entire image, then 
the scheme field does not need to be packed for each list. 

In order to calculate the cost of using a MUX in terms of header sizes for each list, 
5 consider that a 24 bit color image of size 2048 by 2048 needs to be processed in a 
lossy fashion (i.e. the working MUX model of Figure 28). Assume that the size of the 
smallest wavelet transform level is 8 by 8. Thus there are 8 decomposition levels to 
consider in forming the MUX list hierarchy. Assume that the wavelet decomposition 
data has been converted to a 16 bit integer representation. Also assume that there is 
10 only one internal processing scheme for the lists. Given these initial conditions, a 
worst case analysis is conducted to determine the header size for each list. The results 
are tabulated in Table V.l for the Y-Channel wavelet decomposition. 

From table in Figure 36, the total Y-channel header size for this particular image is 
443 bits. A similar calculation can be conducted on the U/V-Channels (assuming one 

15 less decomposition level for each) to yield 368 bits each. The original raw image size 
is 2048 x 2048 x 3 bytes. Thus the MUX packing overhead in this particular example 
is about I header bit for every 854 bits of compressed data. In the lossless case where 
each channels has the same number of decomposition levels (or correspondingly, in 
the 8 bit grayscale case), the overhead is still very small at 1 header bit for every 757 

20 bits of compressed data packed. 

The table in Figure 37 outlines the overhead size for square images with dimensions 
that are a power of 2 beginning at 16 by 16 and ending at 64k by 64k. Note that the 
overhead size if 44 bits in both the lossy and gray scale cases. The reason for this is 
that the wavelet transform is not used on the U/V channels when the original image 
25 dimensions are 16 by 16. The Y-channels is decomposed once. A plot of percentage 
overhead versus image dimension is given in Figure 38. Both lossy and lossless cases 
appear on the same plot. Note that the image size can be quite small while still 
maintaining a relatively low overhead in terms of the total MUX list header sizes. 
Bit Budget control from the MUX Architecture 

30 One of the most difficult tasks that must be addressed in any compression scheme is 
scalability. The internal procedures must be developed such that the image under 
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consideration can be compressed to a user specified size in an optimal manner. The 
lask is compounded for color images since there are three channels to consider. The 
major problem is how to distribute the user specified bit budget between the three 
channels. 

5 Standard color transforms such as YUV or Y1Q are normally used to redistribute the 
RGB color information prior to the multi-resolution decomposition stage of the 
compression procedure. These transformations concentrate most of the energy into 
one channel making them better suited for compression. Another color transform that 
is gaining popularity is the KL transform. This particular transform technique is 

10 based upon a principle component analysis (PCA) implemented on the original raw 
color information in order to determine the optimal color redistribution for the three 
RGB channels. Generally speaking, after completing a color transform stage, two of 
the three channels are down sampled by a factor of 4 to 1 for lossy compression. In 
doing this, the amount of data that must be compressed is reduced by a factor of 2 

15 with minimal loss in visual quality for the reconstructed image. 

The easiest bit budget distribution scheme to implement is one that follows the color 
transform down sampling ratios. If the transform under consideration is YUV-411, 
then 4 parts are allocated to the Y-channel for every 1 part allocated to the U / V- 
channels respectively. However, the energy distribution of the wavelet transform data 
20 generally does not follow this strict down sampling guideline. Instead it varies from 
one image to the next. 

DAC has developed a dynamic bit budget control architecture that is based on the 
implicit information contained in each MUX list. In this Section, a bit budget 
allocation procedure will be introduced that can be used for optimal scalability in 
25 normal processing modes of operation. This technique is based implicitly on the 
amount of energy contained in each color channel of the wavelet decomposition data 
sets. A data ratio concept is used together with the prepack information contained in 
the MUX list structures to determine the budget for each channel. 

Compression procedures that process wavelet coefficients in a bit plane order are 
30 implicitly tracking the energy contained in the decomposition. The most significant 
bits (MSBs) of the largest coefficients are packed first followed by refinement bits 
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and the MSBs that are significant for coefficients at the next bit level. This process is 
repeated in a recursive fashion until the desired compression size is obtained. 

Let y*j i Ug and Vjj , be the total data available in a particular orientation level for the 
three color channels. Then the total amount of data available in the Y-channel 
5 decomposition Y ( is obtained by summing the individual totals. A similar procedure 
can be followed to obtain U ( and V,. 

N 3 

U ' = Un < + 1,1, u * (Eq.V.l) 

N 3 

V,= v N . +S ^ Vij 



20 

The next step is to calculate the pack ratios to be used for each channel of the wavelet 
decomposition hierarchy. The pack ratios are determined by taking the ratio of the 
two largest amounts of data to the smallest amount of data. Let the three pack ratios 
be denoted R y , R u and R v , The smallest data size will be in either the U or V 
25 channels because of the down sampling step. Note that one of the pack ratios will be 
unity. 

Y, 

30 R = 



MINdJpVJ 
V, 

R * " MIN(U„V ( ) 



The pack ratios given in Eq.V.2 are used to determine the optimal amount of data to 
allocate for each color channel based on the user specified compressed file size. 
However, any additional overhead introduced into the final bit stream by the MUX 
must be taken into consideration. Let yh ijt uh;j and vhjj be the individual header sizes 
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in each wavelet channel for a particular orientation level. Then the total header sizes 
for each wavelet channel are obtained by summing the individual totals. 



N 3 
N 3 

10 U h = uh N , + g^uh, (Eq.V.3) 

N 3 

15 



If doing ROI processing, the packing overhead introduced by the mask must be taken 
into consideration in determining the bit budget for each channel. Let the total pack 
20 size for the ROI mask be denoted as M,. The logical approach to take is to make an 
adjustment in the bit budget for each channel based upon the pack ratios of Eq.V.2. 
In this manner, the mask overhead is distributed proportionally to each wavelet 
channel. The unit adjustment factor U a is determined from the total mask overhead 
and the total pack ratio. 

M, 

U. = (Eq.V.4) 

Ry + Ru + Rv 



The adjustment size for each channel is determined by using the result of Eq.V.4 and 
the pack ratios of Eq.V.2. 

Y » = Rv • U. 

U. = Ru . u. (Eq.V.5) 

V * = Rv • V, 



45 A useful set of calculations that comes out of Eq.V.l to Eq.V.5 is the minimum 
compression bits per pixel (BppMin) and the maximum compression bits per pixel 
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(Bpp M „). These values are calculated below (assume that the original raw image file 
size is F, bits). 

BppM« = * ( Y, + U. + V, + Yh + Uh + V h + Y, + a + V. ) 

Ft 

Bpp Min = * ( Yb + Uh + Vh + Y. + U, + V.) (Eq.V.6) 
Fi 

10 



These expression represent bounding compression values available for the image 
under consideration. If all data is packed for each of the wavelet data sets, the result 
15 is Bpp'mm. If just the header information is packed, then the result is Bpp M j n . 

Suppose the user specifies a compressed file size of F oser . The unit bit budget U b b is 
determined from the pack ratios and the total header sizes. 

0m e E-,Y. + U + V0 

Ry + Ru + Rv 



25 Now the optimal bit budget for each channel Y bb , U bb and V bb can be determined 
using Ubb, the individual pack ratios and the overhead parameters. 

30 Ybb = Ubb • Ry + Yh- Y, 

Ubb = U». Ru + U h -U, (Eq.V.8) 
Vbb = V to . R v + Vh - V. 

35 

The expressions given in Eq.V.8 will always yield a near optimal bit budget for each 
color decomposition channel. The technique is based implicitly on the energy 
distribution map recorded in MUX lists and the total overhead associated with the 
underlying process. Note that if processing with no ROI, then that variable falls out 
40 of the calculation. 

This concept can be extended to form pack ratios for each subband if more accuracy 
is required. A similar data ratio technique can be implemented to control the bit 
budget in that case. The necessary information is contained in the MUX list 
structures. However for most practical application, the cited distribution technique is 
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sufficiently accurate. The user specified file size can be obtained within several bytes 
in implementing this technique. In addition, the maximum and minimum attainable 
file sizes are known prior to the final packing stage. 

A simple routine has been developed for distributing the final bit budget in the level 
5 where it is determined it will expire. The idea is quite simple. Given that the bit 
budget will expire on a particular bit plane in a certain transform level, then the bit 
budget is redistributed such that each orientation gets a proportional amount based on 
the amount of data that each orientation can take. The amount of data each 
orientation gets is determined by using ratios between orientations on the bit level 
10 under consideration. This is an important consideration since the net effect is 
approximately a 1 dB improvement in the PSNR. 

Using the MUX for Region Processing 

So far the focus has been on the design of a MUX control architecture for normal 
image processing modes of operation. The main highlights are the data processing 
15 orders, the MUX list structure, the bit budget control technique, and organizing the 
final bit stream. DAC has extended the MUX concept to include a variety region 
processing modes of operation. 

Region Processing Orders 

The processing orders introduced in the previous Section are specific to the normal 
20 processing modes where one or more ROl are not specified. DAC has implemented a 
region processing design based on the natural extension of the MUX concepts 
introduced in the previous Sections. Both lossy and lossless processing orders are 
again considered. 

General Color Region Level Processing Order 

25 The region level processing order places priority on each ROI in a descending fashion 
followed by the normal level priority scheme of the original MUX design. Inherent in 
the MUX scheme is the general assumption that data of similar magnitude in each 
succeeding region is less important to the overall image reconstruction. A secondary 
priority key is placed on the wavelet transform level. Data of similar magnitude 

30 contained in a lower resolution level is more important to the reconstruction 
procedure than data at a higher resolution level. The general processing order for 
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each color channel (assuming a 4 region priority scheme) is given in Figure 39 where 
R ; is the region level for i = 3,2,3,4. 



Region Level Color Processing Order (Lossy) 

The individual orders of Figure V.10 can be combined into a single processing order. 
5 The new order is obtained by interleaving the expressions making use of the inverse 
color and wavelet transformation relationship that is presented for the normal MUX 
modes of operation above. The result is illustrated in Figure 40. 

The intrinsic 4-1-1 color down sampling relationship is maintained for region 
processing MUX modes. In terms of the reconstruction process, the complete level 4 
10 V-channel information (i.e. LL4-HL4-LH4-HH4) is required together with the U and V 
channel low pass information (i.e. LL 3 in each case) for each region under 
consideration. 

Color Interleave Region Level Processing Order (Lossy) 

The color interleave processing order extends naturally to the MUX region processing 
15 modes. The order is illustrated in Figure 41. This type of ordering may be more 
suitable for certain applications that require the data to be encoded for a progressive 
download based on regions of importance. As in the previous case, the down 
sampling relationships that exist for the inverse color and the wavelet space are 
maintained. 

20 Region Color Processing Orders (Lossless) 

As in the lossy color region processing MUX modes, similar processing orders can be 
outlined for the lossless case where full data sets exist for all color channels in each 
region. In this case, corresponding decomposition level orientation information in 
each channel is related in the original color transform space for each region under 
25 consideration. These data are processed and grouped together in the wavelet 
transform space and are ultimately put in the final bit stream by the MUX control 
architecture. 

The processing order for lossless color region processing is illustrated in Figure 42. 
Note that in each case, the full decomposition data set for each channel is maintained. 
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The inherent relationships that exist in the inverse color and wavelet space are 
maintained for each region channel. 

DAC has developed both SNR progressive and resolution progressive MUX modes of 
operation for region processing. In addition, several specialized MUX modes were 

5 developed that may be useful in certain applications. The regions can be defined 
automatically by the technique described in Chapter III, or user defined ROI can be 
used to group the data. User defined ROI can have arbitrary shape or can be defined 
in terms of simple geometric elliptical or rectangular region primitives. 
One of the specialized MUX region processing modes can be introduced at this point 

10 since it fits into the context of the ordering discussion. 

Transparent Region Color Processing Orders (Lossy) 

This particular region color processing order is useful for applications that require 
transparent region channels that has little or no influence on the ordering of the data. 
Either of the normal SNR and resolution progressive modes are overlaid with a 

15 complete region channel description. The technique is implemented simply by taking 
the region index to the inner processing loop in the pseudo code implementation 
example of Figure V.8. This ordering technique may be useful in video processing 
applications where a complete region mask description is required together with a 
compressed frame of information. The region description may be used to process 

20 subsequent frames in the sequence. One of the processing orders that falls into this 
MUX mode of operation is illustrated in Figure 43. In this case, the high bit plane of 
each region list is processed first causing the region classification of all coefficients to 
be transparent to the distribution and packing routines. 

Transparent Region Color Processing Orders (Lossless) 

25 Corresponding transparent region processing modes exist for the lossless case. A 
typical example is given in Figure 44. As in the lossy case, the final bit stream is 
organized independent of the constraints of the region channels which may be useful 
in certain cases. 
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MUX List Structure for Region Processing 

The MUX organizational list structure concepts introduced for normal processing 
modes have been extended to include many region processing modes of operation. In 
this case, the total number of lists is a function of the number of ROI. If there are 4 
5 ROI, then there are 4 times as many MUX lists. However, the basic operation of the 
MUX control architecture is similar in each case. 

There are three high level modes used to categorize the MUX architecture. 

• Normal processing mode (ROI disabled). 

• Region processing mode (ROI only). 

10 • Mixed processing mode (normal and ROI enabled). 

The next Section outlines the operation of the bit budget and MUX controls for many 
of the region processing modes developed at DAC. The mixed processing modes are 
briefly discussed later in the Chapter. 

Bit Budget Control for Region Processing MUX Modes 

15 Transparent Region Level Color Mode 

Transparent region processing is mentioned above. In that particular ordering 
example, the region index is placed in the inner most loop in the bit budget 
distribution function. The processing index placement is basically a method of 
incorporating a transparent region layer of processing into the normal MUX modes 
20 outlined earlier in the Chapter. Processing and packing in this mode gives ROI a low 
priority. 

Distributing in this context ensures that the SNR progressive and resolution 
progressive MUX modes operate as they did before. The exception in this case is that 
there is a variable length region mask overhead that must be taken into consideration. 
25 The bit budget distribution functions are calculated as they were for the normal MUX 
processing modes. Based on the file size (or resolution level) requirements, each 
color channel receives a proportional amount of the bit budget based on the ratio 
distribution technique used in the MUX architecture. 

The DCT auto regions and arbitrary or primitives user defined region types can be 
30 operated in this mode. In DAC's internal processing architecture the user must select 
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the region coverage technique used to categorize the wavelet coefficients. DAC has 
implemented the wavelet mask down sampling technique of the JPEG-2000 VM, and 
it can be used in any of the region processing MUX modes. In this manner a common 
mask can be used in each orientation level or the VM mask down sampling technique 

5 can be used for individual orientation level mask coverage, for both automatic or user 
defined ROI in lossy and lossless MUX modes of operation. The number of region 
categories can be selected as 2, 3, or 4 channels. 

In order to test the transparent mode of operation the YUV-411 color transform is 
applied to a 24 bit 256 x 256 Glacier park mountain scenery image. The 9-7 kernel 

10 implemented in a lifting scheme is used as the wavelet transform. The DCT region 
formation technique of Chapter III is employed to generate the common masks for 
each wavelet transform level. The mean square error (MSE) is measured for a 
number of compression ratios. The result is illustrated in Figure 45 in a plot of MSE 
versus Bpp. The plot shows the break points in MSE for each automatic ROI used in 

15 the example. The rate of image degradation (i.e. MSE and PSNR) must be controlled 
precisely for each ROI. 

The MUX overhead is calculated based on the mask type and process selections. For 
arbitrary user constructed masks, the mask file can be loaded to guide the MUX. In 

20 the 4 ROI common mask case, the overhead is 0.5 Bpp in packing the entire mask. In 
the VM mask case, the overhead is 2.0 Bpp. If simple primitives are used to form the 
user mask instead of an arbitrary shape, the overhead is greatly reduced. The arbitrary 
mask overhead can be reduced with the addition of an entropy coding stage. The 
common DCT mask overhead is cited in Chapter III as approximately 1Kb (0.2 Bpp) 

25 for an 8 bit 256 x 256 image size. These are rough estimates that do not take the 
header sizes into account. However, they do serve as a comparative guideline for the 
current discussion. 

Region Priority Level Color Mode 

In this mode of MUX operation, the pack ratios for each wavelet channel are 
30 determined as in the normal processing mode. However, there is one important 
difference. In this particular mode of operation pack ratios are determined for each 
region channel. This implies that there are 12 pack ratios for a 4 ROI process. The 
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bit budget distribution is based on the overall wavelet channel pack ratios and the ROI 
data totals in each region channel. The distribution function also takes the mask and 
list header size information into account in the overall calculation. 

This mode of MUX operation is designed to distribute the MSE and the PSNR image 
5 reconstruction measurements in an approximately uniform manner for all region 
channels. A proportional amount of overall bit budget is allocated to each region 
channeJ based on the region and color channei pack ratios. Figure 46 illustrates the 
result of using this mode of operation for the Glacier park image. Notice how the 
quality of each region channel degrades in comparison to the others. Auto detected 
10 DCT common masks are used to generate the result. 

Absolute Region Priority Level Color Mode 

In this mode of operation, absolute priority is given to the region channels. The 
distribution function is the same in this case in that the bit budget is divided into 12 

15 according to total data ratio technique mentioned earlier. However the bit budget is 
distributed in a biased fashion giving the highest priority to the most important ROI. 
Some regions may not receive a budget based on the compression requirements. The 
bit budget is distributed region by region beginning at the most important ROI. Thus 
complete sections of the original image can be eliminated altogether if the bit budget 

20 is small. If only the important regions of an image need to be saved, this technique 
can be employed to partition the image. 

A plot of MSE versus Bpp for the absolute region priority color MUX mode is given 
in Figure 47 for the Glacier park image. Notice how the regions fade out very fast 
and sharply. This occurs when the region no longer has any budget allocated to it. At 
25 that point, the region is basically invalidated in terms of any contribution to the 
reconstructed image. That portion of the image basically fades out. Note that the 
amount of quality of the fade out depends on the down sampling technique to translate 
the masks. The VM mask formation technique causes a very gradual . fade out to 
occur. In the common mask approach, the effect is much more abrupt. 

30 Scaled Region Priority Level Color Mode 

In this mode of MUX operation degradation rate of each ROI can be controlled. Each 
ROI contributes to the final bit stream. However, instead of having each ROI degrade 
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at a uniform rate (as Section V.5.3.2.), the quality measurements of each succeeding 
can be controlled. The initial bit budget for each region channel is calculated as 
before in that the specified compressed file size is split into 12 bit budgets to be 
spread between each of the 4 region channels. However after the initial split, the 

5 allocation amounts are changed heuristically based on a priority factor that is set for 
each ROI. For example, suppose it is decided that ROI 1 is 50% more important than 
ROI 2, ROI 2 is 30% more important than ROI 3 and ROI 3 is 20% more important 
than ROI 4 (the background). Using this assumption as a starting point, the amount of 
data allocated to each region channel is changed slightly. The net effect is to cause 

10 the quality measurements in each region channel to degrade at slightly different rates. 
A plot illustrating this effect is given in Figure 48. Note how the MSE break points 
for regions 2 and 3 have shifted slightly towards the left. 

The result of using this particular MUX mode illustrates an important property that 
should be available for any ROI processing technique. The problem with many ROI 
15 processing techniques is that it is difficult to control how much each region 
contributes to the final bit stream. The technique outlined here can be used not only 
to implement this effect, but also to control the degradation for each region channel 
based solely on an importance factor associated with each ROI. 

Region Percentage Priority Level Color Mode 

20 In this mode of operation the user can specify a data percentage for each region based 
on the total amount available in each region channel. The distribution function is 
slightly different in this case. The data totals are determined for each region channel 
along with the wavelet color channel totals. There are still 3 bit budget for each 
region channel. In this case however, the amount of data to include for each region 

25 channel can be set by the user as a percentage of the total in each case. 

Currently when this mode of operation is invoked in DAC's compression engine, the 
user can cycle through the operation until the desired size for each region is obtained. 
The data totals are displayed after each run. Note also that in using this mode of 
operation, one or more ROI can be eliminated or faded heavily as in the absolute 
30 region priority MUX mode of Section V.5.3.3. 
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A plot illustrating the result of using this mode of MUX operation is given in Figure 
49. Initially all available data is included for each region followed by equal 
decrements for each region channel. 

Using the MUX for Mixed Processing 

5 So far both normal (non-region) and ROI processing modes have been discussed. In 
addition to these modes of MUX operation, DAC has developed mixed mode 
processing capabilities. The modes of operation discussed for both normal and ROI 
processing are extended such that they can be run simultaneously for an image under 
consideration, A wavelet transform level partition is conducted based on the desired 
10 number of region and the number of non-region levels. 

Currently non-region levels can be defined for processing lower resolution levels and 
region levels can be defined for process higher resolution levels. However, the 
implementation is not restricted by this distinction. It can be changed to regions over 
non-regions or to an interlaced combination of the two. The parameter that controls 
15 this distinction is termed the region start level. It can be set to any valid wavelet 
transform level (it is set internally to -1 to disable region processing.) Thus the MUX 
technology can be used in exclusive non-region mode, exclusive region mode or a 
combination mixed mode of operation. 

The ordering techniques outlined for normal and region processing MUX modes will 
20 not be duplicated in this Section. The same concepts apply for mixed mode 
processing. The bit budget distribution functions work as they did before, but in this 
case they exist simultaneously. In some modes of operation there may be as many as 
15 bit budget definitions used to organize the final bit stream. The method used to 
determine them has not changes. The same ratio technique that has its basis in the 
25 MUX list structure is used to determine the appropriate allocation in each case. 

There is one additional feature that can be exploited for mixed mode processing. An 
importance factor can be attached to the non-region levels. The net effect of this 
parameter is to taper the amount of data included for the non-region levels. In the 
current implementation, the non-region levels are processed first. Thus they 
30 considered first by the distribution function. The importance factor allows the user to 
decrease the amount of data included for the non-region levels by a certain 
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percentage. The delta amount is considered in the bit budget distribution for the 
region levels. 

The region processing overhead is slightly smaller in this case. Depending on the 
region start level parameter, there will be less region header information required in 
5 the bit stream since there are fewer region levels. However, the header overhead for 
the lower resolution levels is quite small anyway. 

One example is given here to illustrate the operation of mixed mode processing (see 
Figure 50). In this particular case SNR progressive mode is used for the upper 3 
levels and scaled region priority on the bottom 3 levels. Thus the bottom level 
10 degradation rates have been adjusted to favor the most important region and 
attenuating in some fashion between the other 3 regions. Notice how the MSE is 
lower in this case with the same region overhead as the previous case of Figure 48. 

DCT Region Formation as a Classification Scheme 

In observing this result, it is apparent that for the same region overhead, a better result 
15 is obtained in mixed mode. There are a number of reasons for this. The first is that 
the region channel coverage is not an exact overlay. All masks used to group or 
categorize data must deal with the down sampling issue at different resolution levels, 
and the tight overhead restraints of the compression channel. As it appears in the 
results presented here ihe DCT region detection mode can be applied to threshold 
20 wavelet coefficients to form region channels. The amount of data packed for each 
channel can be controlled by the MUX. In addition the initial data split used to create 
the partition can be controlled before the DCT mask procedure begins. 

There is some region migration or intermingling of the coefficients that occurs at the 
boundaries where sharp changes . occur in the original image. Some work was 

25 conducted in developing heuristic techniques to decrease the miscoverage in hot 
areas. And there is a benefit there. Furthermore the DCT low pass filtering stage 
affects the original priority bias used to partition the data. There will be a benefit in 
determining the optimal data split. By adjusting the original region partition (e.g. 
equally spaced regions), the plot of MSE versus Bpp can be set to partition the data in 

30 other ways. Inter-resolution coverage is another problem to consider. The mask 
generation technique of the JPEG-2000 VM addresses the miscoverage problem. 
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DAC has completed some initial investigations of the VM down sampling technique 
and further testing is required. 

There is a trade off between accuracy of each region channel and mask overhead 
required for region channel operation and generation of the accuracy in the first place. 
5 The DCT approach shows much promise especially on the highest resolution level 
where the masks are the most accurate. One of the benefits of using this the DCT 
approach is that it can be translated to other transform levels in the frequency domain. 
Or alternatively, the new masks could be generated at a different wavelet level. 

Other Processing Modes 

10 There are many other modes of region processing operation that have not been 
presented in this document. The whole user defined region generation and MUX 
modes were not included. The processing modes introduced in this Chapter can be 
used in the same way to control each region channel. Simple primitives are cheap in 
terms of overhead. One mode of operation supports full mask sets so that arbitrary 

15 mask can be loaded and sent with the data to form the region channels. The mask 
coverage technique can be selected as common masks (a smaller raw size that is up 
sampled at the decoder side) or VM masks, which retain the original image 
dimension. More time is required to study the effects of both techniques as well as 
other. region formation schemes. 

20 Experimental results for the resolution progressive modes of operation are not 
presented in this document. These modes of operation are not currently available. 
However their implementation is not difficult. These MUX modes are currently 
under development. 

The experimental results presented in this Chapter were obtained using DACs ID bit 
25 level sorting implementation. DAC is currently incorporating the 2D version based 
on EQW into the region processing channel. There may be benefits in retaining both 
techniques. For example, in bi-level image processing. Currently the EQW sorting is 
implemented for normal (non-region) processing MUX modes. Both sorting modes 
are available for lossless compression. There are numerous wavelet kernels and color 
30 transforms for both lossy and lossless compression. In addition, the number of region 
channels can be set with a current maximum of 4. Primitives can be used as desired. 
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The number of primitives is not restricted to 4 with region overlaps given to the most 
important region. 



Bit Stream Syntax 

5 DACs current bit stream structure is rather dynamic given the different types of 
organizational strategies that exist in the underlying core technology. Normal, region, 
and mixed processing modes in addition to arbitrary wavelet, color, entropy coding 
various sorting stages in both lossy and losses cases. 
Currently the core architecture can be divided into 2 categories. 
10 • Lossless compression 

Full data sets lossless color selection, lossless integer lifting scheme wavelet 
types, lossless sorting and packing stages, with additional entropy coding can be 
selected for completely lossless and slightly lossy (some of the color transforms 
are slightly lossy) image compression. 

15 • Lossy compression 

In the lossy case, color selections, more wavelet kernels / lifting schemes, lossy 
sorting and packing stages, with variable length coding can be selected 

In both lossy and lossless cases region and mixed modes can be used. Currently 
normal operational modes can be realized in an almost transparent fashion for many 
20 current compression schemes as well as our own internal EQW / ID sorting. The 
general form for the lossy and lossless header structures is given in the tables in 
Figures 52 and 53. 

The current architecture is very flexible for introducing new technologies. Many of 
the modules are completely interchangeable. The exact bit stream syntax depends 

25 largely on the mode of operation selected by the user (i.e. regions, no regions, sorting, 
lossy/lossless etc.) Generally normal processing modes can be selected for all 
transform levels, or for any number of lower resolution transform levels. Region 
levels can be defined for all transform levels, or the higher resolution levels used in 
combination with the lower resolution normal levels. At some future date, the region 

30 and the normal levels can be mixed. 
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Each unit list organized by the MUX has a header tag. This header lag carries the list 
size and the high bit plane processing level for the list. The current implementation 
uses 5 bytes per list. However bit packing is currently under implementation for tag 
headers. This will reduce this by a significant amount. Only 2 packing schemes are 
5 used at the global level so that no additional tag header information is currently 
required. A diagram illustrating the structure of the tag header is given in Figure 53. 
As other core technologies are added to the core engine or the core technology 
advances, other fields may be required in the tag headers. 

The basic structure for normal modes of operation is given in Figure 54. The diagram 
10 illustrates a lossy pack arrangement for a color image (YUV down sampled color 
transform assumed) according to the packing/processing order given in Figure 30 
above. In this case the code stream consists of the file header, followed by the header 
tags/data. 

15 The basic structure for region processing modes of operation is given in Figure 55. 
The diagram illustrates a lossy pack arrangement for a color image (YUV down 
sampled color transform assumed) according to the packing/processing order given in 
Figure 40 above. In this case the code stream consists of the file header, region 
header, regions description and finally the header tags/data. 

20 The basic structure for mixed processing modes of operation is given in Figure 56. In 
this case the first item in the code stream is. the file header followed by the normal 
processing mode header tags/data. Next in line is the region header, region 
description followed by the header tags/data. Notice that there is a region start level 
parameter V in this case. Since there is one less transform level for U/V channels in 

25 the lossy case, there is a processing shift in the U/V channels. This makes reference 
to the "level split" notion discussed in Chapter V whereby the natural 4:1:1 
relationship that exists for inverse color/wavelet transforms can be maintained for 
internal processing and the final code stream. 
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Transcodability 

In order to illustrate the generality and openness of RICS, two examples are shown to 
depict how the 'transcode' with bi-level images (JBIG) and JPEG can be handled in 
5 RICS. 

JPEG transcode 

Figure 57 shows how a DCT-based code can be produced in RICS. With this 
transcodability, it will be straightforward to write a small conversion program to 
transcode an old JPEG file into a JPEG 2000 file. Although it is possible to transcode 
10 a JPEG 2000 file into a JPEG file (losing some scalability), it will probably be of very 
little use. 
VJB1G transcode 

Figure 58 shows the execution path illustrating how a bi-level image such as a text 
document can be efficiently handled in the RICS system. The NULL transform is 

15 applied (nothing has to be done). Generally binary text documents contain mostly 
high frequency energy. Multi-resolution decompositions will not necessarily be a 
suitable basis for efficient coding. In fact, current well-known techniques specialized 
for binary images are applied directly in spatial domain. While using a RICS system 
for processing a compound document with mixed grayscale/color/text information, a 

20 set of rectangle shapes suffice to enclose the text regions. This is roughly equivalent 
to run-length coding in existing binary image compression techniques to skip strings 
of zeros. The pixels within each rectangular region are then coded into a one- 
dimensional stream via 1-D algorithms or JBIG routines, as discussed above. 

25 Post Processing 

For reconstructed images with wavelet based coding methods at low bit rate, de- 
ringing is usually a useful post processing procedure to improve (mainly) the visual 
quality. Nonadaptive procedures, which apply the de-ringing filtering to all pixels 
without discrimination, have the following problems. 

30 • While they can successfully remove ringing an if acts, they may also wash out 
details of image. 



67 



W0 00/49571 PCT/CAOO/00134 

• They usually rely on too many parameters that are to be determined by human 
users. 

• The process is time consuming. 

The RICS system employs an adaptive de-ringing algorithm for post processing. 
Since the ringing artifacts usually appear around edges where sharp changes occur, 
this adaptive filtering process is applied only to edge areas. As the result, the post 
processing removes artifacts around edges and prevents fine details from being 
smoothed out by the filter. Compared with non-adaptive methods, the processing time 
is remarkably reduced since the filtering is applied selectively to pixels. The diagram 
of Figure 59 shows the algorithm of the adaptive post processing filter. First of all, the 
pixels of reconstructed image are classified into edge pixels and non-edge pixels. 
Edge pixels are enlarged into edge regions since the ringing artifacts do not occur 
right at the edges but around the edges. Then the artifact removal filter is applied 
only to the edge areas. 

We did some tests on this adaptive de-ringing procedure. For the filtering stage we 
used the filter described in the JPEG 2000 VM 3.0 (B). Figure 60 shows the result of 
the modified post processing. Artifacts are clearly visible in the first picture especially 
behind the cameraman's back and around the tripods. As we can see from the second 
picture, artifacts are removed, however, the details in the grass field and on the man's 
pants are also removed. In the last picture, the modified filter successfully removes 
artifacts and at the same time retains the details. Figure 61 shows the edge area (in 
white color) that was used in producing the third picture. 

As the result of using this adaptive filtering, processing time is greatly reduced since 
less pixel are processed. The table in Figure 62 shows the performance comparison of 
the VM 3.0 (B) post filtering and the RICS adaptive filtering 

Edge Detection 

To determine if a pixel is an edge pixel, the average power difference between the 
pixel and its neighboring pixels are measured against a threshold (bottom threshold). 
If the average power difference of a pixel is above this threshold, the pixel is marked 
as an edge pixel. The threshold acts as a high pass filter that filters out the pixels with 
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low power levels (Band pass filters can also be used for edge detection by introducing 
a proper ceiling threshold). 

Edges are thickened to form edge areas after the edge pixels are detected. With the 
right choice of the threshold, this modified filtering process can be applied to 
5 reconstructed images with compression ratios as low as 8:1. Since for most wavelet 
coded images, no artifacts can be visually detected for compression ratio lower than 
8:1, our results suggest that this filter can be applied to most reconstructed pictures 
regardless of its compression ratio. 

Parameters Optimization 

10 One of the problems with Shen's filter is that there are many parameters associated 
with the filter, which makes the filter difficult to use. Reducing the number of 
parameters will improve the usability of the post processing filter. 

There are three different potential functions implemented in VM 3.0 (B) for 
controlling the post processing: Quadratic Truncation, Huber, and Lorenzian. Our test 
15 results shown that the potential functions Huber and Lorenzian do not outperform the 
Quadratic Truncation in any of the 62 cases for PSNR measurements. Furthermore, 
visually Lorenzian creates many noticeable ghost images and shadows, and Huber 
does remove artifacts successfully however it blurs the image more than Quadratic 
Truncation. 

20 

Computation Speed 

For evaluating the three potential functions, the following computation needs to be 
calculate n x (n - 1) 12 times for every pixel (where n = filter size x 2 - 1 ). 

Lorenzian; 1 float division, 2 float multiplication, 1 float addition and 1 

25 float 

log operation 

Huber: 1 comparison, 1 float multiplication or 2 float multiplication 

with 1 

addition 

30 Quadratic Truncation: 1 comparison and 1 float multiplication 



69 



WO 00/49571 



PCT/CA00/00134 



In our tests, Quadratic Truncation is the fastest and it performs the best for artifact 
removal with the least degradation to image quality. Therefore, the other two 
potential functions will not be used. 

Threshold (y parameter in the potential function) 

5 The y parameter is examined using the quadratic truncation potential function {default 
value is 16), given by: 



We can see from the equations that only pixels with similar neighboring pixels are 
10 affected when varying y. When y is decreased, there will be more y 2 as the result of p 
and the intensity of similar pixels output image will be more uniformed resulting a 
more blocky looking type of image (less gradual) for regions with similar intensities. 
Experiments were performed with 2 images (one grayscale and one color image). The 
table in Figure 63 shows the result for the color image. 

15 We can see from the results that the higher the y, the worst the image quality. 
However, the difference in image quality is not very significant. The pictures look 
very similar and there are no significant changes that can be detected. However as 
predicted, the blocky looking clouds (see Figure 60) are seen when y = 8 or below. 
Therefore by setting y lower we can increase the image quality by some small 

20 amounts without changing the visual quality of image. Therefore y = 12 was chosen to 
be the default value for y parameter in the potential function. 

Filter Length 

Filter length (F) determines of the size of samples collected for pixel estimations. It is 
obvious that the larger the filter length the longer the processing time. The default 
25 length given by VM3.0 (B) is 9 pixels. Experiments are performed with varying filter 
lengths to examine how the length affects the image quality (MSE and PSNR) and the 
artifact removal ability. Two pictures are used in this experiment (one color and one 
grayscale). The results are shown in the following tables in Figure 64 and 65. 
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As we can see from ihe experimental results, the small the length, the better the image 
quality and the shorter the processing lime. However, artifacts are not completely 
removed for length = 5 (or smaller) at high compression ratios. The default value for 
the filter length is chose to be 7 pixels to increase image quality and to decrease 
5 processing time without degrading the ability for artifact removal. 

Constraint 

The value of constraint (Thl) affects the filter as described in the equations below. 

y = x + c(d,ThJ) 
d - x~x 

cbLThl) = sign(rf) * Max(0, abs(cO - Max(G\ 2*(abs(rf) - Thl))) 

From the equations above, we can see that as Thl increases, more pixels will have *d' 
as the output for c{d,Thl). Therefore, as Thl increases, more estimate value of x will 
be used as the final estimate and the image quality will decrease while the smoothness 
of an image will increase. In order to keep the image quality as high as possible, Thl 
should be kept as low as possible. However, artifacts might not be properly removed 
if Thl is set too low. Three images were tested with different level of constraints in 
the experiment. The test results are given in the tables in Figures 66 to 68. 

These results show that image quality decreases as constraint increases. However, 
some artifacts (very small, can only be seen when the picture is enlarged) are not 
completely removed when C is equal to 4 or less. In order to achieve the best image 
quality while successfully remove artifacts, the value 8 is suggested to be the default 
value. 

Iteration 

It is obvious that processing time is directly proportional to the number of iterations. 
However, artifacts might not be removed successfully if the image is not processed 
enough times. Figure 69 clearly shows that image quality decreases as the number of 
25 iteration increases. However, when image is enlarged, small artifacts can be still 
visible for Rl. In order to achieve the best image quality while successfully remove 
artifacts, 2 iterations is suggested to be the default setting. 
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Mask Shape 

Different shapes of masks are discussed in Shen's paper, however, only the one with 
the shape of a + sign was implemented in VM3.0 (B). Different masks were 
experimented and the shape of mask is best left unchanged since simplifying the mask 
5 degrades the performance of the filter and complicating the mask increases processing 
time greatly. 

Using the Modified Post Processing Filter 

The adaptive de-ringing is implemented based on the post processing filter in VM3.0 
(B). Few parameters have been eliminated and default values have been established to 
10 increase the usability of the post processing filter. 

To use the modified post processing filter: 

Usage: posi2 -s width height bpp -i infile -o ouif ile (-1 mask lower threshold] 
I-w mask width] f-t thresh] [-f fjengih] [-c constraint] [-r iterations] 

Default values will be used for the parameters if the parameters are not specified in 
15 the command line. The default parameters are: 

mask lower threshold: lower threshold for edge detection [30] 
mask width: width of mask [12] 

thresh: y parameter in the potential function (for estimation) [12] 
fjengih: filter length (7] 
20 constraints: constraints using in the clipping function (8) 

iterations: number of iterations \2] 

The modified post processing filter performs well with the above default parameters, 
however, these parameters can be changed at the command line if the user wishes. 
The post processing filter can now be applied to pictures with 8:1 compression ratio 
25 with very small increase in MSE. The modified post processing filter seems to 
improve image quality for compression ratio beyond 11:1. 

In practice, there are two ways for the user to play with the settings: on the encoder 
side and on the decoder side. 

Decoder end 

30 The decoder will have the full control of the post processing and the of the associate 
parameters. The encoder will have no control on how the images will look at the 
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decoder end. Accordingly, there will be no addition to file header since no post 
processing parameters will be included. 



Encoder end 

When necessary, the encoder can also predetermine the post processing filter 
5 ■ parameters. Post processing filter parameters will be stored in the image header, and 
the image will be restored according to these parameters. In this setting, the user at 
the encoding end knows exactly how the image will look at the decoder end. 
However, adding these parameters in the header will increase the size of the 
compressed image. 

10 Total of 6 parameters will need to be packed in the header: mask lower threshold, 
mask width, estimation threshold, filter length, constraints and number of iterations. 
The table in Figure 70suggests range limits on the parameters to reduce the header 
size. 

In total, three bytes will be needed to -include these parameters in the header. 
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WE CLAIM 

1. A region-based method for encoding and decoding digital still images to 

produce a scalable, content accessible compressed bit stream comprising the 
5 steps; 

decomposing and ordering the raw image data into a hierarchy of multi- 
resolution sub-images; 

10 determining regions of interest; 

defining a region mask to identify regions of interest; 

encoding region masks for regions of interest 

determining region masks for subsequent levels of resolution; and 



15 



20 



scanning and progressively sorting the region data on the basis of the 
magnitude of the multi-resolution coefficients. 

2. An apparatus for the region-based encoding and decoding of digital still 
images that produces a scalable, content accessible compressed bit stream 
comprising: 

25 a means of decomposing and ordering the raw image data into a hierarchy of 

multi-resolution sub-images; 

means of determining regions of interest; 

30 means of defining a region mask to identify regions of interest; 

means of encoding region masks for regions of interest; 

means of determining region masks for subsequent levels of resolution; and 

35 

a means for scanning and progressively sorting the region data on the basis of 
the magnitude of the multi-resolution coefficients. 

3. A region-based system for encoding and decoding digital still images that 
40 produces a scalable, content accessible compressed bit stream and comprises 

the steps: 

decomposing and ordering the raw image data into a hierarchy of multi- 
resolution sub-images; 

45 

determining regions of interest; 
defining a region mask to identify regions of interest; 
50 encoding region masks for regions of interest 
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determining region masks for subsequent levels of resolution; and 

scanning and progressively sorting the region data on the basis of the 
5 magnitude of the multi-resolution coefficients. 

4. A method for encoding and decoding digital still images to produce a scalable, 
content accessible compressed bit stream comprising the steps: 

10 decomposing and ordering the raw image data into a hierarchy of multi- 

resolution sub-images; 

setting an initial threshold of significance and creating a significance index; 
15 determining an initial list of insignificant blocks; 

forming the list of significant coefficients by encoding a significant map using 
a quadtree representation; 

20 recursively reducing the threshold values and repeating the encoding process 

for each threshold value; and 

transmitting refinement bits of significant coefficients. 

25 5. An apparatus for encoding and decoding of digital still images that produces a 
scalable, content accessible compressed bit stream comprising: 

a means of decomposing and ordering the raw image data into a hierarchy of 
30 multi-resolution sub-images; 

means for setting an initial threshold of significance and creating a 
significance index; 

35 means for determining an initial list of insignificant blocks; 

means of forming. the list of significant coefficients by encoding a significant 
map using a quadtree representation; 

40 a means of recursively reducing the threshold values and repeating the 

encoding process; and 

transmitting refinement bits of significant coefficients. 

45 6. A method of decoding digital still images to produce a scalable, content 
accessible compressed bit stream comprising the steps: 

decoding the bitstream header; 
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determining the initial threshold values and the array of initial significant 
pixels, insignificant bits and wavelet coefficients; 

decoding the significance maps; 

5 

modifying the significance lists and decoding the refinement bits for each 
threshold level; 

reconstruct the wavelet coefficient array; 
perform the inverse wavelet transform; and 
reconstruct the image. 

15 7. A method of transmission of digital signals that creates a scalable, content 
accessible bitstream comprising the steps: 

pack the most significant bits of the largest coefficients first followed by 
refinement bits and the most significant bits that are significant for coefficients 
20 at the next bit level; 

repeat this process in a recursive fashion until the desired compression size is 
obtained; 

calculate the pack ratios to be used for each channel of the wavelet 
25 decomposition hierarchy by taking the ratio of the two largest amounts of data 

to the smallest amount of data; 

determine the optimal amount of data to allocate for each color channel based 
on the user specified compressed file size; and 

30 if performing region of interest processing, consider the packing overhead 

introduced by the mask when determining the bit budget for each channel. 
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Shape 


Coding Parameters 


Availability in RICS 


Rectangle/Square 
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present 
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Region M ask Dynamic Object Extracted Dynamic Regions 
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Mask Spectrum Size 
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Y-channel: LU-HL 4 -LH4-HH4-HL 3 -LH3-HH3-HL2-LH2-HH 2 - HL,-LH r HH, 
U-channel: LL 3 - HL 3 -LH 3 - HH 3 - HL Z - LH 2 - HH 2 - HL r LH,-HH, 
V-channel: LL 3 - HL 3 -LH3-HH r HL2-LH 2 -HHrHLrLH r HH, 
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Ylw-Yhw-Ylh^-Yhh^Ulu-Vllt 

YHU-YLHJ-Y H HrUHU-ULHJ-UHHJ-VHU- V LHr V KHr 
YHL2-YLH2-Y H H2-UHU-ULH2-UHH2-VHU- V LH2-VHHr- 
YHLrYLHrYHHI-UHLI-ULHrUHHI-VHLj-VLHrVHHI 
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Ylu-Yhw-Y L h4-Y H h4-Uuj- ,v uj- 

YhU-UhL3* V HU-Y L H3-UlH3- V LH3-YhH3-UhHJ- V HH3- 
Y HL2- U H L2- V HL2" Yui2" Uui2" V U I2- YhH2" U H H2- V HH2* 
YHU-UHLI-VHLrYLHl-ULHl-VLHI-YHHrUHHI-VHH) 
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Y L W-Uu^-V U ><-YHl><-Y L H4-YHHrUHL4-Uui4-UHH-- V HL4-V LH 4-VHH4- 

Yhl3-Ylh3-Yhh4-Uhu-Ulh<-Uhh4-Vhu- v lh3- v hh3- 

Y H L2-YlH2-YhH2-UhU-Ulh2-UhH2-V H l2-VlH2- V HH2- 

Yhli*Yi>|]-Yhhi"Uhli*Ulhi~Uhhi~Vhli"Vlhi"^hhi 
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Yu4- Ulu- Vlu- Y H u- U hl *- V H l4- Yum-Uu^- V liw - Y HH 4-U H h4- V HH4 

YHU-UHU-VHU-YLHrUuu-VL^-YHHj-UHHJ-VHHa 
YhU- UhLT V H L2- Yun- U|_H2"^LH2~ Y HH2 - U H H2" V HH 2 
YKLrUHLrVHLI-YLHi-ULHi-VLHi-YHHj-llHHI-VKH) 
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Data Type: MUXLIST 
Parameters: 

liTotByiesPacked - long integer total byies packed into the data buffer for this list. 
cScheme - character processing scheme used for data contained in this list. 

cHighBit - character highest bit-level where data processing begins for this list. 

♦pucMuxBuff - pointer to unsigned character buffer where data for each 

bit-level is packed for this list 

Fields for MUX: information for packing after list processing is complete. 

pliBitPackInfo[16] - pointer to long integer number of bits packed into the data buffer 

at each bit-level for this list 
JiCurBytesCount - long integer current byte count used for bit budget distribution 

when packing this list. 
cCurBitLevel - character current bit-level used for packing this list. 

cReroainingBits - character remaining bits to be packed at a given bit-level when 

data to be packed is not evenly divisible by 8 for this list. 
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CALCULATE Channel BitBudget // determine optimal bil-budget for each color channel. 
INITJALIZE Channel CunrentBiiPlane //highest bit pJane thai exists in each color channel. 
INITIALIZE liCurBytesCoum and cRemainingBiis FOR each MUX list 

FOR Each Color Channel // process each channel separately. 

WHILE Channel BitBudget > 0 AND Channel CurremBilPlane >=0 

FOR Each Wavelet Transform level // beginning at lowest resolution level. 

FOR Each Orientation Set of Data //according to lossy case natural processing order. 
IF cCurBitlevel NOT_EQUAL to Channel CurrentBitPlane 

CONTINUE 
ELSE 

SET BitLevelByies to pliBitPacJdnfo[Channe\ CurrentBitPlane] » 3 
SET RemBits to pliBitPacklnfo [Channel CurrentBitPlane] &. 7 
IF SvmicRemainingBits, RemBits) >= 8 
INCREMENT BitLevelByies by 1 
DECREMENT cRemainingBits by 8 - RemBits 
■ ELSE 

INCREMENT cRemainingBits by RemBits 
ENDIF 

IF Channel BitBudget >= BilLevelBytes 

INCREMENT UCurBytesCount by BitLevelByies 
DECREMENT Channel BitBudget by BitLeyelBytes 
DECREMENT cCurBitUvel by 1 

ELSE 

INCREMENT UCurBytesCount by Channel BitBudget 
SET Channel BitBudget to 0 
ENDIF 
ENDIF 

IF Channel BitBudget EQUALS 0 

BREAK 
ENDIF 
END FOR 
END FOR 

DECREMENT Channel CurBitPlane by 1 
END WHILE 
END FOR 
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Y-channel: R,(LU-HL.-LH 4 -HH. 1 .HL 3 -LH r HH3-HL 2 -LH r HH 2 - HL r LH,-HH,). 

R z (LL J ,-HL,-LH4-HH4-HL 3 -LH 3 -HH r HLrLHrHH2- HLpLH.-HH,), 
R3 (LL< -HWLIVHHr HL 3 - LH 3 -HH 3 -HL 2 - LH 2 -HH 2 - HL r LH r HH,) ( 
R,(LL,-HL4-LH,-HH,-HL 3 -LH3-HH r HL2-LH 2 -HH 2 - HL r LHrHH,). 

U-channel: R,(LL r HL r LH r HH r HL 2 -LH r HH r HL,-LH,-HHi), 
R 2 (LL r HL 3 -LH 3 -HH 3 -HL r LH 2 -HH 2 - HL r LH r HH,) ( 
R 3 (LL r HL 3 -LH 3 -HH 3 -HL 2 -LH 2 -HH r HL,-LH,-HH,), 
R^(LL 3 - HL 3 -LH r HH 3 -HL r LHrHH r HL r LH,-HH,). 

V-channel: R,(LL r HL 3 -LH r HH r HL r LH r HH r HL r LH,-HH,), 
R2CLL3- HL 3 -LH 3 -HH r HL 2 -LH 2 -HH 2 - HL r LH r HH,), 
R 3 (LL 3 - HL r LH 3 -HH r HL 2 -LH r HH r HL,-LH,-HH,). 
R4(LL 3 - HLj-LH r HH r HL 2 -LH r HH r HL.-LH.-HH,). 
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YHU-YLHrYHHa-UHU-UuO-UHHB-VHIJ-VLHj-VHHj- 
YKU-Yui2-Y H H2-UHU-ULH2-U m i r VHL2-VLH2-V H H2- 
YHLrYLHI-YHHrUHLrULHj-UHHi-VHLI-VLHi-VHHl), 

R 2 (Y1X4- Y H L<- YiM4- Y H h4-Uuj- Vuj- 

Yhu- YlH3- YhH3" UhlJ"Ulh 3 -Uhh3* V HU" V LH3- V HH3" 
YHL2"YLH2-Y H HZ-UHU-UtH 2 -UHH2-VHL2- V LH2-VHH2- 

YHL!-Y LHr YHHrUHLrULHrU HH! -V HL ,-V LHl -VHHi), 
R3(Ylu-Yhw-Y LH4 -Yhh4-Ulu- v uj- 

Yhu-Y L h3-Y H h3-Uhu-U L h3- u hh3-Vhu-Vlh3-Vhh3- 

YhL2"YlH 2 -YhH2"UhL2"UlH2~UhH2 - VhL2-Vl>|2-Vhh 2 - 

Yhli - Ylh , - Y HH 1 -U H li -^imi - U H h r V H li - Vlh 1 - V H m ) . 
R4 ( Ylu- Yhu- Yu«* Yh wUjjj- Vhj- 

YHU-Y U j3-Y H H3-U K u-Uui3-UHH3- V HU- ,V U)3-VHH3- 
YHL2-Y U n-Y H H2*t)HL2-U L H 2 -UHH2-VHU-VLHrVHnr 
YHLI-YLHl-YHHt-UHLI-ULHrUHHl-VHLrVLHt.VHHiJ. 
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Ri (Yur YHirY L H < -Y H H4-U LL r Vlu- 

YHU-UHU-VHU-Yuu-ULHJ-VLHj-YHHj-UHHa-VHHJ- 
YHU- U HLrVHLrYLH2-ULH2- V LH2-Yffi|2-UH!l2-VHHr 
YHLrUHLl-VHLt-YLHrULHrVLHrYHHI-UHHrVHHl). 

^(Yux-Yhw-Ylh.-Yhh^-Uuj-Vuj- 

YHU-UHU-VHU'YlJtJ-ULW-VtHj-YHlirUHJD-VHW- 
YhL2*U HU-^HLT YlH2'U u«- V LH2" Y HH2- U 1 1HT V H H2* 

Y H LrUHLrVHLrYLHrUun-VLHi-Y H ni-UHH]- V HHi). 

^(YuM-YHW-YLHi-YHH^ULU-Vuj- 

Yhl3-U H u-V H u-Ylh3-Ulh3-Vui3-Yhh3-Uhh3-Vhh3- 
Yhu-U hu~ v hu- Ylh2-U mr ^lhz- Y hhz- U ma- Vhht 

YHL1-UHLr V HLrYLHrULHrVLHrYHHrUHH|-V H Hl). 

R.(Y LL 4-Yhl4-Y LH 4-Yhh--U L u-Vuj- 

YHLrUHu-Vmj-Yuo-Uuo-VLHrYHHJ-UHHrViiHs- 

YHL2-UHL2-V H LrYui2-ULH2-VLH2-YnH2-UHH2-VHH2- 
YHLrUHLI-V H Ll-YLHrULHr V LHrYHHrUHHr V HHl)- 

FIGURE 41 



RHYu^-Ulu-V^-Yhw-Yuhj-Yhm^Uhw-Ulh-i-Uhw-Vhw-Vlh-i-Vhim- 

Y H U-YuU-YHH4-UHbrULH4-UHH<- V HU* V LH3- V HH3- 

YHu-Yuu-YHHZ-UHu-Uua-U^-VHu-VuirVHHj- 
YHLi-YLHi-YHHrUHLrUun-UHH.-VHu-VLHrVHHi). 

R2(Y LU -U| JM -V Lbr Y, IL rY LW -YHtM-UHW-ULH4-UHH--V|iL4-VuM-VHHr 

Y HI j-Y ll H3-YHH4-UHLrUu M -UHHrVHLrV L HrV|iH3- 

YHUrYLH2-Y H H2-UHL2-ULK2-UHH2- V HL2- V LH2-VHH2- 
YHLi-YuH-YHHrUilLrULHrUHHrVHLI-VLHt-VHHl). 
RSfYu^-Uu^-Vu^-YHU-YLH^-YHW-UHW-UuM-UHH^VHW-VLH-l-VHIM- 

Yhl3-Ylh3-Yhh4-Uhu-Ulh4-Uhh4-Vhu-Vlh3-Vhh3- 

YHU-YLH2-Y H H2-UHL2"ULHrUHH2- V HL2- V LH2- V HH2- 
Y|| L i-Y L H|-YHHrUHLl-U L H 1 -UHHr V HLr V LHrVHHl).- 

R4(Y u ^-U lw -V ll< -Yhl4-Y L h J -Y„h4-U hlj -U L h4-Umh4-Vhl4-V L h J -Vhh4- 

YHU^YuU-YHH^UHwULH^UHHrVHU-VuirVHHJ- 
Y HL2" YlH2~ YhH2* ^hL2-UlH2" ^HH2" V HL2~ ^lh2"^HH2" 
YHLrYLH1-YHHrUHL1-ULH1-UHHt- V HLl-VLH1- V HHl)- 
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Yll4-Yhl4-Ylh4-Yhh4-Uuj-Vll3- 

YhL3-YlH3-YhH3-Uhl3-UlH3-UhH3-Vhu-VlH3-V H H3- 
YHL2-YLH2-YHH2-UHLrU LH2 -U H H2-VHU-Vi,H2-VHHr 

YHLrYuirYHHi-UHLi-UtHi-UHHrVHu-VLHrVHHi (Rl. R2. R3 , R4). 
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YLU-ULU-V lX ,-y Ku .Y L H4-YHH4-UHL--U L H4-UHHi-V H Li-V LKj( -VHH4- 

Yhu-Y L h3-Yhh<-Uhlj-U LH) i-Uhh4-Vhu- v lh3-V HK 3- 

YHU-Y L H2-YHH2-UHU-U L H2-U H H2-V H u-V L H2-VHHr 
YHU-YLHI-YHHI-UHLl-ULHrUHHrVHLl-VLHrVHH, (Rl, R2. R3, R4) 
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FIGURE 44 

MSE vs. Bpp for Glacier Park Image 





Region 3 












Region 4 — 












— * 
























1 




i 







0.5 



1.0 1.5 
Bpp 



2.0 



2.5 



FIGURE 45 




FIGURE 46 



20/35 



WO 00/49571 



PCT/CAOO/00134 



MSE vs. Bpp for Glacier Park Image 



w 



1000 
800 
600 
400 
200 
0 



















V^. - R 


sgion A 






— Region 3 






Region ! 

N 


3 












1 A 







0.0 0.5 



1.0 1.5 
Bpp 



2.0 2.5 



FIGURE 47 



24 
20 
16 
12 
8 
4 



MSE vs. Bpp for Glacier Park Image 



Region 














_ Region 4 


















^, Region 2 





























0.0 0.5 



1.0 1.5 
Bpp 



2.0 



2.5 



FIGURE 48 



21/35 



WO 00/49571 



PCT/CA00/00134 





FIGURE 50 



22/35 



WO 00/49571 



PCT/CAOO/00134 



Lossless Case 



Packed Item 


bits 


Selections / Comments 


header size 


8 ■ 


Total Header is normally about 6 bytes 


8 bit grayscale / 24 bit color 


1 




Image Width 


10 


(64-1024 columns) 


Entropy Coding 


2 


None, Basic, Adaptive 


Image Height 


10 


{64-1024 rows) 


Wavelet Transform Levels 


3 


Level >= 1 


Reeion Channels 


2 


1.2, 3,4 


Mask Type 


2 


None, User, Auto fXJCT) 


Pack Raw Mask Flag 


1 


Read in file for mask 


Mask Procedure 


3 


DCT Common Mask. Raw Common Mask with VM Translation, 


Down Sampling Type 


2 


Heuristic DCT Down Sampling 


Region Start Level 


4 


No Regions to # wavelet levels 


Lossy Flag 


1 


Lossless 


Color Transform Type 


2 


2 Internal, Y1Q (full data sets) 


Sort Type 


1 


ID, EQW 


Wavelet Kernel 


2 


Lifting Scheme 



FIGURE 51 



23/35 



WO 00/49571 




PCT/CA00/00134 

Lossv Case 


Packed Item 


bits 


Selections 


header size 


8 


Between 6 to 12 bytes 


8 bit grayscale / 24 bit color 


1 





Image Width 


10 


(64-1024) 


Image Height 


10 


(64-1024) 


Wavelet Transform Levels 


3 


L>=2 


Region Channels 


2 


1.2,3.4 


Mask Type 


2 


None. User, Auto (DCD 


Pack Raw Mask Flag 


1 


Read in file for mask 


Mask Procedure 


3 


DCT Common Mask, Raw Common Mask with VM Resolution 
Translation, User primitives Common or Full Si2e. User arbitrarily defined 
in Common and Full sizes 


Down Sampling Type 


2 


Heuristic DCT Down Sampling 


Region Start Level 


4 


No Regions to #wavelet levels 


Lossy Flag 


1 


lossy 


Color Transform Type 


2 


2 Internal, KL, YIQ, YUV (down sampled data sets) 


Post Filter 


1 


Under Implementation 


Sort Type 


1 


ID. EQW 


Wavelet Kernel 


3 


Daubaches, Symlet, Coiflet, Biorthogonal. Lifting Schemes 


Filler Type 


4 


Various common filter types/sizes 


KL Color Transform 


48 


(if enabled) 



FIGURE 52 



Total Bytes Packed in List 



1 




( 
p 


^ore Comprs! 
rocess List (r 

On 

1 


ion Scheme I 
lot currently i 

linal Uncomf 
(not currcn 

H 
I 

r i 


Jseto 
n use) 

ire&sed Lisi Size 
ily in use) 

gh Bit- Plane Level Where 
'rocessing Begins for Lisi 

r 










§ t 9 Packed Data List 



FIGURE 53 



24/35 



3NSDOCID; <WO 00*957 1«A> 



WO 00/49571 



PCT/CAOO/00134 



File Hetuter 



TigfY-LU Tag (Y-HLj Tag (Y-UU Tag (Y-HH.) 

(III 

I 





Tacked Data 




Packed Data 




Packed Datt 




Packed Data 


Tag (U-LU4) 

J 


Tag (V-LU.) 

J 


Tag (Y-HU.) 

J 


Tag! 

J 


Y-LH„i) 




Packed Daia 




Packed Dato 




Packed D*tt 




. Packed Dau 



Tag (U-HH,) Tag(V-HL,> * Tag (V-LH.) Tag (V-HH.) 



I • I 



Packed Dau 



FIGURE 54 



25/35 



WO 00/4957J 



PCT/CAO0/O0J34 



File Header 



Region Header 



Ref ton Docriptkm 



Region 1 

TagfY-LLJ TagfY-HU TagfY-UU Tag (Y-HHJ 





Packed Data 




Packed Data 




Packed Data 




Packed Data 


Tag 

4 


(U-LU,) 


Tag (V-LUJ 

1 


Tag (Y-KUJ 

4 


Tag (Y-LH^d 

4 




Packed Data 




Packed Data 




Packed Data 




Packed Dan j 


Tag 

j 


(IAHHJ 


Taj fV.HU) * 


Tag {V-LHJ 

4 


T.g 

4 






Packed Data 




Packed Data 




Packed Data 




Packed Data 



Region 2 

TagfY-LU T.g<Y-HU Tag fY-UU Tag CY-HHJ 

4 





Packed Data 




Packed Data 




Packed Data 




Packed Data 


Ta| 

4 




Tag (V-LUi) 

4 


Tag (Y-Hl»i) 

4 


Tag 

4 






Packed Data 




Packed Data 




Packed Data 




Packed Oau 


T«g 

4 


(U-HHi) 


Tag (V-HU) • 

4 : 


Tag CV-LH.) 

4 


Tog 

4 


(V-HHJ 




Packed Dou 




Packed Data 




Packed Data 




Packed Data 



FIGURE 55 



26/35 



VSDOCID. <WO_0O«B57iA2J_> 



WO 00/49571 



PCT/CA00/00134 



Ti ( (Y.LU 



1a t fY-HU 



Tag (V LtU 



• Tbi fY-HHJ 





Packed Dau 




Packed Dan 




Packed Daia 




Packed Data 


1 


i 


Taj (Y-Hl-i) 

I 


Tat (Y-LH-,) 

i 




Paeked Dau 




Packed Dau 




Packed Dau 




Packed Daia 


1»t (U-MUtl Tag (V-HU-J 


Taj IV-UU 

1 


i 




Packed DaU 




Packed Dau 




Packed Dau 




Packed Data 



Region Header Rcfwn Description 



Region 1 

Ta| (Y-HUJ Tag fY-LH^*) Taj (Y-HJCm) 



i i A 




Packed Dau 




■ Paeked Data 




Packed Daia 


Taj (U-HU**) 

I 


Ta« W-LH^J 


Taj <U-HU»J 




Packed Dau 




Packed Data 




Packed Data 


Tai {V-HU-J 


Taj (V-LH~4> 


Tag (V-HK-J 


1 

T 














Packed Data 




Packed Dau 




Packed Data 


Taj (V-HL.) 


Taj (V-HU 


Taj (V.HKi) 




Packed Data 




Packed Dau 




Packed Data 


Ta ( (Y-HU..,) 


Region 2 
Taj tY-UL-.) 


Taj (Y-HH^J 


1 

T 














Packed Dau 




Packed Dau 




Packed Daia 


Tag (U-HU-J 

i 


Taj (U-UtJ 


Tag (U-HH«,,> 




Packed DaU 




Packed DaU 




Parked Daia 


Ta| (V-Hl i) 

1 


Ta ( <V.Ui~J 


Tit <V-HH^1 




Packed Dan 




Packed Dau 




Packed Data 


Tij (V-HU 

i 


Taj (V-HL,) • 


Taj (V.HH,) 




Packed Dau 




Paeked DiU 




Packed Dau 



FIGURE 56 



27/35 



WO 00/49571 



PCT/CAOO/00134 




0£ 

a 

« 



CiO 

toil 
B 
*£ 

o 

o 

s 

B 

o 
no 

a 

a 
PC 











"« 
w 




<? 




to 








<-» 




o 




2 




1 

ft! 




o 


i 







3 

s = 

.£ p 



3 

c 

•A .£ .2 « 
go 8 | 

« P c £ 

C < &H 



g 8 

© v « 

o B * 

Oh • O 

"-9 U 



28/35 



NSDOCI& *WO_00*9571A2J_> 



WO 00/49571 PCT/CAOO/00134 



-3 







E 
























a 






a 






k. 






H 






d 












D 






2 



73 

CO 

o 



i 



bl) 
& 




00 
IT) 



a 



2 H 

Is 



5 i/fl 

»JL « o W 

io s i 
"Mis 

05 i < &- 



o a « 
U9 m 

ft d 



29/35 



WO 00/49571 



PCT/CAOO/00134 



Reconstructed 
Image 



Edge 
Detection 



Edge Pixels 



Edge 

Thickening 



De-ringing 
Filtering 



Final 
Image 



FIGURE 59 




Re-constructed Image 



VM 3.0 (B) Post-Processing 

FIGURE 60 



Adaptive Post-Processing 




FIGURE 61 



30/35 



1NSDOCID: <WO 00*9571A2_I_> 



WO 00/49S71 



PCT/CA00/00134 



Compression 
ratio 


VM 3.0 (B) 
time (sec) 


RJCS 
Adaptive 
lime isecj 


Compression, 
ratio 


VM 3.0 (B) 
• time (sec) 


R1CS 
Auapuve 
time (sec) 


3.21 /U 


/J1U 




3.709 


22.392 


5.398 


9.U14U 


7 ?/\0 

/ .L.O\J 


1 702 


8.889 


23.023 


8.151 


40.210 






10 000 


22.442 


5.288 


48.763 






1 1 428 


22.232 


5.809 


56.492 


8.262 


1.402 


13.333 


22.462 


5.878 


66.550 


8.703 


1.382 


16.000 


22.482 


5.629 


81.143 


6.769 


1.362 


20.001 


22.522 


5.489 


103.716 


7.171 


1.175 


26.666 


22.382 


5.568 


130.168 


7.170 


1.222 


32.000 


22.422 


5.338 


169.019 


8.310 


1.132 


40.002 


22.352 


3.896 


253.337 


7.130 


1.052 


79.987 


22.282 


3.465 



FIGURE 62 



Compression 
Ratio 


PSNR (r=8) 


PSNR(r=12) 


PSNR(Y=16) 


3.709 


31.329611 


31.177579 


30.511348 


4.000 


31.300452 


31.154471 


30.490295 


5.333 


31.157031 


30.994670 


30.357978 


8.000 


30.630383 


30.474574 


29.862022 


8.889 


30.401649 


30.216825 


29.646734 


10.000 


30.067778 


29.909074 


29.375402 


11.428 


29.691347 


29.546710 


29.056083 


13.333 


29.228454 


29.103298 


28.645196 


16.000 


28.407724 


28.276664 


27.908937 


20.001 


27.508670 


27.413509 


27.134810 


26.666 


26.385419 


26.307257 


26.084268 


32.000 


25.333808 


25.280108 


25.111808 


40.002 


24.696655 


24.660851 


24.509536 


79.987 


22.179274 


22.152787 


22.093103 


100.004 


21.592595 


21.568550 


21.525140 



FIGURE 63 



31/35 



WO 00/49571 



PCT/CAOO/00134 



Compression 
Ratio 


PSNR (F=3) 


PSNR (Fr5) 


PSNR (F=7) 


PSNR (F=9) 


PSNR (F=ll) 


1.583 


35.514592 


34.206867 


33.687448 


33.409849 


33.223806 


1.600 


35.516204 


34.201801 


33.674565 


33.408178 


33.220934 


2.000 


35.314595 


34.080237 


33.538998 


33.301232 


33.090358 


2.667 


34.957612 


33.765719 


33.291270 


33.011713 


32.832969 


4.000 


34.002797 


32.997735 


32.564248 


32.338257 


32.200522 


8.000 


31.089735 


30.554027 


30.324381 


30.196463 


30.106473 


8.889 


30.572145 


30.142348 


29.935209 


29.796335 


29.725453 


9.999 


30.131952 


29.795631 


29.604549 


29.472403 


29.394763 


11.429 


29.629346 


29.390456 


29.261471 


29.13131 


29.064764 


13.334 


28.854873 


28.701271 


28.602145 


28.527964 


28.470308 


16.000 


27.870101 


27.760478 


27.671142 


27.632609 


27.579655 


19.999 


27.031806 


26.910914 


26.857123 


26.835492 


26.794267 


26.662 


25.962692 


25.994100 


25.999759 


26.019787 


25.991909 


40.010 


24.159134 


24.178761 


24.173011 


24.191618 


24.182049 


80.020 


21.654340 


21.683920 


21.731559 


21.761676 


21.774576 



FIGURE 64 



Compression 
Ratio 


PSNR (F=3) 


PSNR (F=5) 


PSNR (F=7) 


PSNR <F=9) 


PSNR (F=1I) 


3.709 


31.656960 


30.740420 


30.806267 


30.531348 


30.298514 


4.000 


31.634763 


30.735842 


30.791791 


30.490295 


30.278695 


5.333 


31.436411 


30.558898 


30.635321 


30.357978 


30.143580 


8.000 


30,819587 


30.065789 


30.126713 


29.862022 


29.680912 


8.889 


30.537579 


29.839074 


29.879854 


29.646734 


29.474362 


10.000 


30.170406 


29.541099 


29.603527 


29.375402 


29.205815 


11.428 


29.740858 


29.186726 


29.255123 


29.056083 


28.880456 


13.333 


29.255225 


28.752598 


28.839294 


28.645196 


28.494299 


16.000 


28.391715 


27.982943 


28.075866 


27.908937 


27.788083 


20.001 


27.479317 


27.162369 


27.270921 


27.134810 


27.032976 


26.666 


26.329007 


26.079132 


26.175590 


26.084268 


25.999266 


32.000 


25.279960 


25.088255 


25.181387 


25.111808 


25.044080 


40.002 


24,649334 


24.486896 


24,568579 


24.509536 


24.446939 


79.987 


22.195351 


22.099853 


22.127757 


22.093303 


22.066329 


100.004 


21.598939 


21.520845 


21.561808 


21.525340 


23.501686 



FIGURE 65 



32/35 



INSDOCID: <WO 004957iA2_l_> 



WO 00/49573 



PCT/CA 00/00134 



Comp 
Ratio 


MSE C4 


PSNR C4 


MSE C6 


PSNRC6 IMSEC8 
| 


PSNR C8 


MSE CIO 


PSNR CIO 


1.583 




39.229159 




36.588355 ^^gjggg 


35.063845 




34207692 


1.600 




39219220 




36.586070 


35.053114 




34.193764 


2.000 


ms&m 


38.925459 




36.370297 j^F^^| 


34.883178 




34.055766 


2.667 




38266669 




35.957748 


34.513929 




33.737744 


4.000 




36.451122 




34.709662 g§gggl|5§& 


33.535544 


mam 


32.932853 


8.000 j 


mmk 


31.648852 






30.684031 


mum 


30.416851 


8.889 




30.904531 




30.521456 Jgj^|6gg|g 


30.181222 




29.977944 


9.999 


nam 


30.303783 




30.009204 


29.756500 




29.605845 


11.429 




29.727687 




29.511321 ^^^^^ 


29.350576 




29232327 


13.334 




28.962695 


^*!*^^§^? 




28.688748 


ohsbb 


28.602798 


16.000 




27.899311 




27.785245^^g^i^| 


27.717939 




27.659899 


19.999 




26.946559 


wmmm 


26.886839 ^^Pj^ 


26.848426 




26.832480 


26.662 




25.926574 




25.949079 'ffi^&km 


25.980794 




25.999188 


40.010 




24.153977 




24.165936 




24.181593 




24.187081 


80.020 


iSi 


21.704967 




21.719818 


mmm 


21.736217 




21.748404 



FIGURE 66 



Comp 
Ratio 


MSE CI 2 


PSNR CI 2 


MSE CM 


PSNR CI 4 1 MSE CI 6 


PSNR CJ6 


1.583 




33.409849 


MS 


32.656521 fellSI 


31.965423 


1.600 




33.408178. 




32-656995 ge4j3gg£2g| 


31.962959 


2.000 




33.301232 




32.537766 ^^^^ 


31.860463 


2.667 




33.011713 




32.302688 gg§g£?ll| 


31.697984 


4.000 




32.338257 


mwrnm 


31.758905 


mmmm 


31.252020 


8.000 




30.196463 




29.942129 




29.737452 


8.889 




29.796335 


mmmm 


29.626910 




29.463474 


9.999 




29.472401 




29.329176 


^8t8986i§^ 


29.160108 


11.429 




29.131310 




29.012885 




28.869385 


13.334 




28.527964 




28.436833 




28.340178 


16.000 




27.632609 




27.597400 




27.545611 


19.999 


8UHB 


26.835492 




26.821589 




26.803981 




26.019787 




26,013378 




26.005148 


80.020 ^^^^^ 


24.191618 i 




24.183297 




24.165350 




21.761676 




21.758578 




21.747777 



FIGURE 67 



33/35 



BNSDOCID- <WO 0fUf)fi7iA? i > 



WO 00/49571 



PCT/CAOO/00334 



Comp 
Ratio 


MSE C6 


PSNR C6 


MSE C8 


PSNR C8 


MSE CIO 


PSNR CIO 


3.709 




32.017177 




31.280383 




30.858253 


4.000 




31.988238 




31.261918 




30.837691 


5.333 




31.777734 




31.077137 




30.684074 


8.000 




31.092049 




30.513768 




30.167911 


8.889 




30.781017 




30.245934 




29.939214 


10.000 




30.403643 




29.923100 




29.633349 


11.428 


d ^*8S3H85E 


29.947021 




29.546056 




29.295539 


13.333 




29.462736 




29.086581 




28.861458 


16.000 




28.546036 




28.254853 




28.078653 


20.001 




27.596921 




27.398874 




27.261184 


26.666 




26.433597 




26.285258 




26.181599 


32.000 




25.362805 




25.252939 




25.175851 


40.002 




24.710196 




24.626109 




24.569853 


79.987 




22.238546 




22.176299 




22.133956 


100,004 


>flHBR 


21.634422 




21.590113 




21.557097 



FIGURE 68 



Comp 
Ratio 


PSNR Rl 


PSNRR2 


PSNR R3 


PSNR R4 


PSNRR5 


1.583 


36.574570 


34.642943 


33.409849 


32.697519 


32.241311 


1.600 


36.562052 


34.637526 


33.408178 


32.690100 


32.238019 


2.000 


36.362537 


34.486701 


33.301232 


32.599597 


32.146102 


2.667 


35.932465 


34.134055 


33.011713 


32.347816 


31.926978 


4.000 


34.722586 


33.301598 


32.338257 


31.778517 


31.416868 


8.000 


31.264379 


30.640440 


30.196463 


29.892131 


29.673122 


8.889 


30.660431 


30 A 83697 


29.796335 


29.547117 


29.365451 


9.999 


30.201141 


29.800439 


29.472401 


29.244209 


29.090109 


11.429 


29.706536 


29.398255 


29.131310 


28.942109 


28.805907 


13.334 


28.946950 


28.733637 


28.527964 


28.378407 


28.263955 


16.000 


27.924645 


27.784119 


27.632609 


27.510373 


27.424183 


19.999 


27.018210 


26.938891 


26.835492 


26.749663 


26.687918 


26.662 


26.025142 


26.045564 


26.019787 


25.971153 


25.921551 


40.010 


24.209514 


24.214338 


24.191618 


24.152663 


24.116273 


80.020 


21.70436 21.747276 


21.761676 21.762267 


21.751451 



FIGURE 69 



34/35 



WO 00/49571 



PCT/CA00/00134 



Parameter 


Number of Bits 


Value Range 


Mask Threshold 


7 


0-127 


Mask Width 


4 


5-20 


Estimation Threshold 


4 


5-20 


Filler Length 


3 


3-10 


Constraints 


4 


3-18 


Iteration 


2 


1-4 



FIGURE 70 



35/35 



(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) World Intellectual Property Organization 
International Bureau 




(43) International Publication Date (10) International Publication Number 

24 August 2000 (24.08.2000) PCT WO 00/49571 A3 



(51) International Patent Classification 7 : G06T 9/00, 9/40 

(21) International Application Number PCT/CA00/00134 

(22) International Filing Date: 15 February 2000(15.02.2000) 

(25) Filing Language: English 

(26) Publication Language: English 

(30) Priority Data: 

2,261,833 15 February 1999(15.02.1999) CA 

(71) Applicant (for all designated States except US): DIG- 
ITAL ACCELERATOR CORPORATION [CA/CA]; 
Suite 840, 650 West Georgia Street, Vancouver, British 
Columbia V6B 4N9 (CA). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): WANG, Meng 
[CA/CA); 3027 Laurel Street, Vancouver, British Co- 
lumbia V5Z 3T6 (CA). VANG, Xue, Dong [CA/CA]; 
3446 Rideout Bay, Regina, Saskatchewan S4S 7C3 (CA). 
SIMON, Brent (CA/CA); 210 West Keith Road, North 
Vancouver, Briush Columbia V7P 1Z3 (CA). QU, Li 
[CA/CA); 318-165 East 6tb Street, North Vancouver, 
British Columbia V7L 1P2 (CA). XIONG, Yi [CA/CA]; 
328 Louis Riel House. SFU. Vancouver, British Columbia 



V5A 1S6 (CA). WONG, Michael [CA/CA]; 7831 Welsley 
Drive, Burnaby. Briush Columbia V5E 3X4 (CA). 

(74) Agent: MBM & CO.; P.O. Box 809, Station B, Ottawa, 
Ontario K1P5P9(CA). 

(81) Designated States (national): AE, AL, AM, AT. AU, AZ, 
BA, BB, BG, BR, BY, CA, CH, CN, CR, CU. CZ. DE, DK, 
DM, EE, ES, Fl, GB, GD, GE, GR GM, HR, HU, ID. IL, 
IN, IS, JP, KE, KG, KP, ICR, KZ, LC, LK, LR, LS, LT, LU, 
LV, MA, MD, MG, MK MN, MW, MX, NO, NZ, PL, PT, 
RO. RU. SD, SE. SG, SI. SK, SL, TJ, TM, TR, TT, TZ, UA, 
UG, US. UZ, VN, YU, ZA, ZW. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS, MW, SD. SL, SZ, TZ, UG, ZW), Eurasian patent 
(AM. AZ. BY. KG, KZ, MD, RU, TJ, TM), European patent 
(AT, BE, CH. CY. DE, DK, ES, Fl, FR. GB, GR, IE, IT, LU, 
MC, NL, PT, SE), OAPI patent (BF, BJ. CF, CG, CI, CM, 
GA, GN, GW, ML, MR, NE, SN, TD, TG). 

Published: 

— With international search report. 

(88) Date of publication of the international search report: 

5 April 2001 

For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations" appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



(54) Title: METHOD AND SYSTEM OF REGION-BASED IMAGE CODING WITH DYNAMIC STREAMING OF CODE 
BLOCKS 




Transform 



Region 
Definition 



Intra-Region Coding 



< 



On 
O 

o 



Shape Code 



Entropy Code 



Code stream 



MUX 



□ 



(57) Abstract: The invention is a new effective and fast method and apparatus for still image compression, transmission and decom- 
pression. The present invention implements optional sorting and encoding methodologies to create object-oriented, shape coding or 
region coding in still image and video coding that is content accessible, scalable and computationally efficient The invented, mul- 
tiplexed transmission regime ensures a robust, scalable and content accessible bit stream and includes a dynamic, bit budget control 
architecture. 



INTERNATIONAL SEARCH REPORT 


Intt. 1 Application No 

PCT/CA 00/00134 




I A. CLASSIFICATION OF SUBJECT MATTER " — 

IPC 7 G06T9/G0 G06T9/40 




' 


According lo International Patent Ctasstficaiion (IPC) or to bom national classification and IPC 
B. FIELDS SEARCHED 






y-jmum DOCU ™^ seafCrtea ictossrlicaiion system followed by classical symbols) 



i the fields searched 



!. search terms used) 



t passages 

TUBMAN D: "JPEG 200G Verification Model 
VM3A, IS0/IEC JTC1/SC29/WG1 N1143" 

I SO/I EC JTC1/SC29/WG1: CODING OF STILL 
PICTURES - JBIG.JPEG, 
1 February 1999 (1999-02-01), pages 1-89 
XP002146G99 M y * 

cited in the application 
sections VIII-4.5, IX-5, XI 

WO 00 04721 A (QU LI ;SIM0N BRENT (CA); 
WANG MENG (CA); YANG XUE DONG (CA); DIGIT) 
27 January 2000 (2O00-01-27) 
claims 1,15,29 

US 5 563 960 A (SHAPIRO JEROME M) 
8 October 1996 (1996-10-08) 
column 2, line 47 - line 67 



| | X| Further documents are listed in the continuation of box C. 
I * Special categories erf coed documents : 

*A* document defining the general stale of the an which is not 
conudered » be of particular relevance 

T" earlier document but pubfesned on or after the international 
filing date 

| V obcutneni which may throw doubts on priority daimfs) or 
which is cued to establish the publication date of another 
atation or other special reason (as specified) 

~CT document referring to an oral disclosure, use. exhibition or 
other means 

"P* document published prior to the international fifing, date but 
later man the pnorny date claimed 

Date of the actual completion cl the international search 

36 August 2006 



1-3 



1-3,6 



fX~l Patent family members axe fested in annex. 

T tzler documerrt published after the rmernational fUtno dale 
or pnonry date and not in contfcet with the application but 
cried to understand the principle or theory underlying the 
invention 

"X" document of particular relevance: the claimed rrventJon 
cannot be considered novel or cannot be considered to 
involve an inventive step when the document is taken alone 

"V document of particular relevance; the claimed nventjon 
cannot be considered to involve an inventive step when the 
document is combined with one or more other such docu- 
ments, such aDrntoina&on being obvious to a person skilled 
rn the art. 

'** document member of the same patent fa miry 
Date of mailing of the intemational search repon 

1 5. It 00 



t Name and mailing address of the 6A 

European Patent Office. P.B. 5818 Patentfaan 2 
NL 2280HVflijsw* 
Tel. (+31-70) 340-2040. Tx. 31 651 epo nl. 
Fax: (+31-70) 340-3016 

Form PCT/tSA/210 (secono sheet) (Xtf? 1932) 



Authorized officer 



Bouchaala, N 



«DOCID:<WO_ 00*8571 A3 I > 



page 1 of 2 



INTERNATIONAL SEARCH REPORT 



biter. U Application No 

PCT/CA GO/00134 



^Continuation) DOCUMENTS CONSIDERED TO BE RELEVANT 



Category * Citation ot document, with indication, where appropriate, of the relevant passages 



Relevant to daim No. 



P,X 



TAUBMAN D: 'High performance scalable 
image compression with EBCOT' 
PROCEEDINGS 1999 INTERNATIONAL CONFERENCE 
ON IMAGE PROCESSING (CAT. 99CH36348) , 
PROCEEDINGS OF 6TH INTERNATIONAL 
CONFERENCE ON IMAGE PROCESSING (ICIP'99), 
KOBE, JAPAN, 24-28 OCT. 1999, pages 
344-348 vol.3, XP900921211 
1999, Piscataway, NJ, USA, IEEE, USA ISBN: 
0-7803-5467-2 . 

page 345, right-hand column, line 29 - 
line 46 

FRAJKA T ET AL: "PROGRESSIVE IMAGE CODING 
WITH SPATIALLY VARIABLE RESOLUTION" 
PROCEEDINGS OF THE INTERNATIONAL 
CONFERENCE ON IMAGE PROCESSINGS, LOS 
ALAMITOS, CA: IEEE, 1997, pages 53-56, 
XPOG0792714 ISBN: 0-8186-8184-5 
abstract 
sections 2, 3 

EGGER 0 ET AL: "ARBITRARILY-SHAPED 
WAVELET PACKETS FOR ZEROTREE CODING" 
IEEE INTERNATIONAL CONFERENCE ON 
ACOUSTICS, SPEECH, AND SIGNAL PROCESSING - 
PROCEEDINGS. (ICASSP) , US, NEW YORK, IEEE, 
vol. CONF. 21, 1996, pages 2335-2338, 
XP000681695 ISBN: 0-7803-3193-1 
sections 2, 3 



4,5 



1-3,6 



1-3.6 



form PCT/1SV210 (oortto»jalion of tecond SK<1> (Juty ^2) 



page 2 of 2 



INTERNATIONAL SEARCH REPORT 



II itional application No. 
PCT/CA G6/0O134 



Box I Observations where certain claims were found unsearchable (Continuation of item 1 of first sheet) 



This International Search Report has not been established in respect oi certain claims under Article 17(2)<a) tor the following 
1. | | Claims Nos.: 

because they relate to subject matter not required to be searched by this Authority, namely: 



reasons: 



2. Q Claims Nos.: 



becaus ^ e >' relate 10 Pfr?° f *»» International Application that do not comply with the prescribed requirements to such 
an exient thai no meaningful International Search can be carried out, specifically: 



3. Claims Nos.: 

because they are dependent claims and are not drafted in accordance with the second and third sentences of Rule 6.4(a). 

Box II Observations where unity of invention is lacking (Continuation of item 2 of first sheet) 

This International Searching Authority found multiple inventions in this international application, as follows: 



1 . P] As all required additional search fees were timery paid by the applicant, this International Search Report covers at) 
» searchable claims. 



2. Q As aO searchable claims could be searched without effort justifying 
of any additional fee. 



an additional tee, this Authority did not invite payment 



3. I~| As only someot the required additional search fees were timery paid by the applicant, this International Search Report 
1 — 1 covers only those claims tor which fees were paid, specifically claims Nos.: "epon 



4 - 0 fh^^inn fiS ^tl^l^ ST* ^ a « ),icanl - Consequent, this Internal Search Report rs 

restncted to the invention first mentioned m the claims; h is covered by claims Nos.: 



1-6 



Remark on Protest 



| | additional search fees were accompanied by the applicant's protest. 
| | No protest accompanied the payment of addrtiona! search fees. 



Form PCT/ISA/210 (continuation of first sheet (1)) (Jury 1 998) 

WSDCCID: <WO 0CM957iA3J_> 



INTERNATIONAL SEARCH REPORT 



International Application No. PCT/CA 0O/GO134 



FURTHER INFORMATION CONTINUED FROM PCT/ISA/ 210 



1. Claims: 1-6 

region-based embedded coding and decoding of a 
wavelet- transformed still image, coding single wavelet 
subband with an embedded quadtree 



2. Claim : 7 

in the embedded compression of a still image, allocating the 
bit budget to each color channel based on a user defined 
file size 



INTERNATIONAL SEARCH REPORT 
^Hlormation on patent tamlly members 



Patem document 
cited in search report 



Publication 
date 



(men il Application No 

PCT/CA 09/09134 



Patent famfty 
members) 



WO 0004721 
US 5563960 



A 
A 



27-01-2800 
08-10-1996 



Publication 
date 



AU 
NONE 



4597699 A 



07-02-2000 



c D ^ t/K: * /,,n "— — ' tfy mnti) (July 1992) 

JMSOOCID <WO 0O49571A3J > 



(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



CORRECTED VERSION 



(19) World Intellectual Property Organization 
International Bureau 

(43) International Publication Date 
24 August 2000 (24.08.2000) 




PCT 



(10) International Publication Number 

WO 00/49571 A3 



(51) International Patent Classification': G06T 9/00, 9/40 

(21) International Application Number: PCT/CAOO/00134 

(22) International Filing Date: 15 February 2000 (15.02.2000) 

(25) Filing Language: English 

(26) Publication Language: English 

(30) Priority Data: 

2.26 1 ,833 15 February 1 999 ( 1 5 .02. 1 999) C A 

(71) Applicant (for all designated States except US): DIG- 
ITAL ACCELERATOR CORPORATION ICA/CA]; 
Suite 840, 650 West Georgia Street. Vancouver, British 
Columbia V6B 4N9 (CA). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): WANG, Meng 
[CA/CA]; 3027 Laurel Street, Vancouver, British Co- 
lumbia V5Z 3T6 (CA). VANG, Xue, Dong [CA/CA]; 
3446 Rideout Bay, Regina, Saskatchewan S4S 7C3 (CA). 
SIMON, Brent [CATCA]; 210 West Keith Road, North 
Vancouver, British Colombia V7P 1Z3 (CA). QU, Li 
[CA/CA]; 318-165 East 6th Street, North Vancouver, 
Briiish Columbia V7L 1P2 (CA). XIONC, Yi [CA/CA]; 
328 Louis Riel House, SFU, Vancouver, British Columbia 
V5A 1S6 (CA). WONG, Michael [CA/CA]; 7831 Welsley 
Drive, Bumaby, British Columbia V5E 3X4 (CA). 



(74) Agent: MBM & CO.; P.O. Box 809, Station B, Ottawa, 
Ontario KIP 5P9 (CA). 

(81) Designated States (national): AE, AL, AM, AT, AU, AZ, 
BA, BB, BG, BR, BY, CA, CH, CN, CR, CU. CZ, DE, DK, 
DM, EE, ES, FI, GB, GD. GE, GH, GM, HR, HU, ID, IL, 
IN, IS, JP, KE, KG, KP, KR, KZ, LC, LK, LR, LS, LT. LU, 
LV, MA, MD, MG, MK, MN, MW, MX, NO, NZ, PL, PT. 
RO, RU, SD, SE, SG, SI, SK, SL, TJ, TM, TR, TT, TZ, UA. 
UG, US, UZ, VN, YU, ZA, ZW. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS, MW, SD, SL, SZ, TZ, UG, ZW), Eurasian patent 
(AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European patent 
(AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, 
MC, NL, PT, SE), OAPI patent (BF, BJ, CF, CG, CI, CM, 
GA, GN, GW, ML, MR, NE, SN, TD, TG). 

Published: 

— With international search report. 

(88) Date of publication of the international search report: 

5 April 2001 

(48) Date of publication of this corrected version: 

21 June 2001 

(15) Information about Correction: 

see PCT Gazette No. 25/2001 of 21 June 2001, Section II 

f Continued on next page J 



(54) Title: METHOD AND SYSTEM OF REGION-BASED IMAGE CODING WITH DYNAMIC STREAMING OF CODE 
BLOCKS 



< 



Image 
Data 




Transform 




Region 
Definition 




Intra-Region Coding 










♦ 






i 


f 












Shape Code 






Entropy Code 





Code stream 



MUX 



o 



(57) Abstract: The invention is a new effective and fast method and apparatus for still image compression, transmission and decom- 
pression. The present invention implements optional sorting and encoding methodologies to create object-oriented, shape coding or 
region coding in still image and video coding that is content accessible, scalable and computationally efficient The invented, mul- 
tiplexed transmission regime ensures a robust, scalable and content accessible bit stream and includes a dynamic, bit budget control 
architecture. 



BMSOOCIO:*WO MW9E71A3 IA> 



WO 00/49571 A3 lOIIIIIIIIIIIIIllDNlllWllllinillDII 



For rwo-letier codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations" appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



SNISDOCID: cWO OCX957iA3.IA> 



WO 00/49571 



PCT/CA00/00134 



METHOD AND SYSTEM OF REGION-BASED IMAGE CODING WITH 
DYNAMIC STREAMING OF CODE BLOCKS 

5 Field of the Invention 

The present invention relates generally to image coding, and more particularly to the 
compression and streaming of scalable and content-based, randomly accessible digital 
still images. 

10 Background of the Invention 

With the rapid growth of the internet and multimedia applications, there is a great 
demand for new image coding tools that that will provide for high quality processing 
capability, an efficient internal architecture, and flexibility in terms of future 
technological advances. This is a challenge that has been put forth before the JPEG 
15 2000 Committee. AJthough the main focus should be to provide state-of-the-art 
compression performance, JPEG 2000 should also offer unprecedented content based 
accessibility of the compressed format to support applications of various needs. It is 
highly preferred and advantageous that image content be accessed, manipulated, 
exchanged, and stored in compact form. 

20 In order for JPEG 2000 to be the standard coding foundation of new generation image 
processing systems, it must provide for efficiency in coding, different types of still 
images (bi-level, gray-level, color) with different characteristics (natural images, 
scientific, medical imagery, rendered graphics, text, etc.) within a unified system. In 
addition to providing low bit rate operation with quality performance superiority to 

25 existing standards, this new system should include many modem features as listed in 
the JPEG 2000 requirement document. 

The open architecture and the set of algorithms presented in this document are based 
on Digital Accelerator Corporation's (DAC) Region based Image Coding System 
(R1CS). RJCS has not only achieved a rale distortion performance competitive with 
30 best known compression techniques, but also demonstrates a high degree of openness 
and flexibility that will accommodate most well known algorithms as well as new yet 

J 
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to be implemented. The features supported by RJCS covers almost all those listed in 
JPEG 2000 Requirements document. 

Generality 

Generality is a primary concern in the architectural design of RICS. The RICS 
5 architecture attempts to be an integrated platform that supports and facilitates a 
variety of applications that may have different characteristics and requirements. For 
example, providing efficient lossless and lossy compression in a single code stream; 
efficient processing of compound documents containing both bi-level (text) and color 
imagery; progressive transmission by pixel accuracy and/or by resolution; random 
10 access of arbitrary shaped regions; and so on. 

Openness 

We also understand that the new JPEG 2000 standard is intended to be a dynamic, 
rather than static, suite of coding tools that support the new generation imagery 
applications and, at the same time, keeping abreast with the progress of technology. 
The mathematical foundation for those leading candidates of new image coding 
methods (most noticeably the various types of multi-resolution analysis techniques, 
such as wavelet transforms) is relatively young and still under intensive investigation. 
If an architectural design is based on or restricted to one or several particular existing 
coding methods, it may become outdated very quickly. 

In attempting to produce a flexible and open platform, the RICS architecture 
organizes the image coding process into a set of functionally separable modules, such 
that each module can be developed and optimized individually. Furthermore, all 
modules in the system are designed to be functionally orthogonal with each other, in 
the sense that a new algorithm, without affecting the functionality of other modules, 
25 can effectively replace the base algorithm of any specific module. 

This open architecture will not only be able to accommodate future new algorithms, 
but also makes compatibility with other standards an easy and natural extension of 
many concepts, as will be explained later in this document. 
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Accessibility 

Content based accessibility is becoming an important feature in supporting 
applications such as multimedia database query, internet server-client interaction, 
content production, remote diagnostics, and interactive entertainment. The content- 
5 based accessibility requires that semantically meaningful visual objects be used as 
basis for image data representation, explanation, manipulation, and exchange. With 
images being represented in compressed format, it is desirable to perform retrieval 
operations directly in the code space without requiring image reconstruction. In fact, 
any search algorithms that require image reconstruction will be infeasible from a 
10 practical viewpoint because of the huge amount of images in most image databases. 
The previous JPEG and many existing coding techniques focus primarily on the issue 
of compression ratio, paying minimal attention to the need of content based image 
retrieval. 

RICS, by its name, has a fundamental consideration to various types of regions in 
15 images. The regions referred to as ROI (Region of Interest) are usually user-specified 
primitive geometric shapes, such as rectangles or circles. The regions that define the 
visual objects are usually of arbitrary shapes. Regions can also be generated as the 
result of certain mathematical operations or transform properties (e.g. significance of 
transformation coefficients) used to partition the image into disjoint regions of various 
20 (e.g. tiles, hierarchical blocks, or arbitrary shapes). 

In designing the RICS system, DAC considered carefully the distinction between the 
concept of an object and that of a region. RICS is open to very general definition of 
region. A region is a 2D spatial identity with pure syntactic contribution to a code 
stream. In contrast, an object may contain semantic information. Therefore, the 
25 region is perceived as a more elementary and reliable description than the object. 
RICS is region based, not object based. RICS supports a rich set of region types, 
from primitive geometrical shapes, tiles, hierarchical blocks, to most general arbitrary 
shapes. 

The region based coding strategy effectively supports the content-based accessibility 
30 of imagery data. Specifically, this ability enables random access to the code stream as 
well as providing a processing channel for user defined regions of interest or tile 
based techniques. Supporting MPEG-4 object based accessibility is one of the main 
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objectives of the RJCS design. Furthermore, region-based coding provides a natural 
bridge for 'transcodability' with JPEG and JBIG. 

Scalability 

Many applications require image data that is available at different resolutions or 
5 qualities for decoding. For example, in a progressive transmission process, the bit 
rate control mechanism should allow the image data to be transmitted in certain 
priority order, and be able to truncate the remaining data flexibly, either upon request 
from the receiving terminal or upon channel limitation. Scalable image coding 
involves generating a coded representation (code stream) in a manner that facilitates 
10 the reconstruction of the image at multiple resolutions or quality levels by scalable 
coding. 

Ideally, the control of scalability should be centralized in a single module as the last 
stage on the encoder side right before the code stream is fed to the communication 
channel. Furthermore, it is desirable that the scalability control module can 
15 completely handle the required processing locally, without any further request 
propagating back to any previous stages in the encoder. In this way the scalability 
control module avoids the need for multi-pass computation. 

The RICS architecture is designed to support three types of scalability: scalability in 
terms of pixel precision, spatial resolution, and regions. 

20 Compactness 

There is no doubt that the new image coding standard should offer a higher 
compression performance than the former JPEG, especially at the low bit rate end. 

Integrating the compactness, scalability, and accessibility into a general purpose, 
flexible, and open architecture represents a challenge for JPEG 2000. The RICS is 
25 designed to provide a solution. The basic idea of region based coding is as follows: 

• The input image data, after certain transformation, becomes a set of image 
primitives. This set of primitives can be wavelet transform coefficients, DCT 
coefficients, other transform coefficients, or even raw image data. 
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• The image primitives are grouped into regions. Region definition can come from 
user defined ROI, from other application modules, or from certain automatic 
segmentation algorithms running in the primitive space. 

• Each region contains one or more independent coding units (ICU). The primitives 
in an ICU are encoded and decoded independently, without reference to primitives 
of any other ICU. This procedure is called the intra-region coding. The outcome 
of an ICU operation is a code block. 

• A multiplexer (MUX) is employed to integrate the code blocks into the final code 
stream. 

A Brief Description of the Drawings 

Figure 1 is a simplified RICS block diagram. 
Figure 2 is a detailed RICS block diagram. 
Figure 3 is subband decomposition schemes. 
Figure 4 is a Wavelet Filter Library. 

Figure 5 is a performance comparison of YUV and standardized color systems. 

Figure 6 is geometric shapes supported by RICS. 

Figure 7 is a hierarchical partitioning. 

Figure 8 is the relationship between regions and objects. 

Figure 9 is threshold masks obtained from Level 1 Lena Wavelet Decomposition. 
Figure 10 is a Level 1 common mask obtained by thresholding the combined data set. 
Figure 1 1 is individual region masks obtained by separating raw the common mask. 
Figure 12 is individual region masks obtained by separating raw common mask. 
Figure 13 is a spectral magnitude of zigzag spectrum as a function of position. 
Figure 14 is common mask spectrum filter sizes and captured coefficient numbers. 
Figure 1 5 is a coefficient banding concept used to quantize the common mask 
spectrum. 

Figure 16 is Quantization Band Sizes for Common Mask Spectrum of size 128 by 
128. 

Figure 17 is Spectral Quantization Band Sizes for Other Common Mask Sizes. 
Figure 18 is a Mask Overhead for Grayscale Images. 

5 
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Figure 19 is a low pass Common Mask Approximation for general coefficient 
classification. 

Figure 20 is a Mask Quantization following the Spectral Content for Bit AJlocation. 
Figure 21 is Translated Common Masks for Remaining Resolution Levels. 
5 Figure 22 is the three types of 1CU structures. 

Figure 23 is a Pavement of a region using type-1 ICUs. 
Figure 24 is a Embedded quad tree flowchart. 
Figure 25 is VLC Codes for EQW. 
Figure 26 is the transcode with JBIG. 
10 Figure 27 is the 'trans-out* and 'trans-in* modes for transcoding with JBIG. 
Figure 28 is Wavelet Mallot Decomposition for Lossy Color Transform Data. 
Figure 29 is Level Priority Processing Order for Each Channel (Lossy Case). 
Figure 30 is Level Priority Processing Order (Lossy). 
Figure 31 is Level Priority Color Interleave Processing Order (Lossy). 
15 Figure 32 is Level Priority Color Processing Order (Lossless). 

Figure 33 is Level Priority Color Interleave Processing Order (Lossless). 
Figure 34 is Typical MUX List Data Structure. 
Figure 35 is SNR Progressive Bit Budget Distribution Scheme. 
Figure 36 is MUX List Overhead for Y-Channel 8-LeveI Decomposition. 
20 Figure 37 is MUX List Overhead for Square Images of Various Sizes. 
Figure 38 is MUX List Overhead for Square Images of Various Sizes. 
Figure 39 is General Region Level Priority Color Processing Order (Lossy). 
Figure 40 is Region Level Priority Color Processing Order (Lossy). 
Figure 41 is Color Interleave Region Level Priority Processing Order (Lossy). 
25 Figure 42 is Region Level Priority Color Processing Order (Lossless). 

Figure 43 is Transparent Region Level Priority Color Processing Order (Lossy). 
Figure 44 is Transparent Region Level Priority Color Processing Order (Lossless). 
Figure 45 is a graph representation of the transparent region level in color 
mode for the 4 DCT region channels. 
30 Figure 46 is a graph representation of the region priority level in color 
mode for the 4 DCT channels. 

Figure 47 is a graph representation of the absolute region priority level 
color mode over the 4 DCT region channels. 
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Figure 48 is a graph representation of scaled region priority level color 
mode of the 4 DCT channels. 

Figure 49 is a graph representation of the region percentage priority level 
in color mode over the 4 DCT region channels. 
5 Figure 50 is a graph representation of the mixed processing multiplexer mode 
of operation over the 4 DCT channels 
. Figure 5 1 is a diagram of the basic lossless header structure. 
Figure 52 is a diagram of the basic lossy header structure. ( 
Figure 53 is a diagram of the basic header tag structure. 
10 Figure 54 is a block diagram of the basic bit stream syntax for normal modes 
of operation in the lossy case. 

Figure 55 is a block diagram of basic bit stream syntax for region modes of 
operation. 

Figure 56 is a block diagram of the basic bit stream syntax for mixed modes 

15 of operation. 

Figure 57 is a block diagram of the JPEG transcode/decode method 
Figure 58 is a block diagram of the JBIG transcode/decode method 
Figure 59 is a block diagram of the modified post filtering procedure 
Figure 60 are three gradations of post processing in digital still images.. 

20 Figure 61 is a representation Edge areas where the de-ringing filtering is 
selectively applies. 

Figure 62 is a table of Test flmage: Camera (256x256 grayscale) Test Image: 
hk (256x256 color) 

Figure 63 is a table of Test Image hk.raw results 
25 Figure 64 is a table of Test Image camera.raw (grayscale) results 

Figure 65 is a table of Test Image hk.raw (color image)results 

Figure 66 is a table of Camera raw (part a) results 

Figure 67 is a table of Camera.raw (part a) results 

Figure 68 is a table of HK raw results 
30 Figure 69 is a table of: Quality versus iterations. 

Figure 70 is a table of: Quality versus iterations. As discussed. 
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A Detailed Description of the Drawings 

The System Architecture 

The architecture of the RICS system can be described as the scheme of dynamic 
5 streaming of code blocks. In this design, the image primitives of ICUs generate unit 
code blocks. A code block is a logically independent unit of coding and decoding 
which does not rely on information contained in any other blocks. Compression is 
achieved in each block coder, and the coding efficiency of the block coders 
determines the overall efficiency of a RJCS system. The openness of the system is 

0 reflected in the different coding algorithms that can be used to produce different code 
blocks. The system uses a multiplex structure (a MUX) to assemble the code blocks 
into the code stream and realize the bit rate allocation. In short, the RICS architecture 
allows for flexibility in the areas of compactness and openness to the block coders 
and, at the same time, allowing the multiplexer to handle the various schemes of 

1 scalability and random accessibility to the code stream. 

It should be noted that the block used in RICS is a logical unit rather than a geometric 
concept. A code block may correspond to an 8x8 tile (in the case of JPEG mode 
coding), a pyramid data structure in zero tree like coding schemes, a rectangular area 
in block based coding schemes, or an arbitrary shaped area in the raw image buffer. 
The independence of encoding and decoding is the primary requirement of a code 
block. A code block is the outcome of an intra-region coding. 

Figure 1 illustrates the simplified RICS coding architecture. A more detailed diagram 
is shown in Figure 2. A detailed description of the function of each module is given 
in this document. Because the RICS is intended to be an open architecture, DAC does 
not limit each module to any specific algorithm. Instead, any individual algorithm 
can be placed into a module as long as it meets the functional requirement of that 
module. Supporting algorithms are included for certain modules, which reflect 
preferred operation of the overall system. 
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Types of Input Image 

As shown in Figure 2, the RICS architecture supports the coding of three types of 
image data: grayscale, color, and bi-level images. 

Transformations 

5 In a typical multi-resolution coding scheme, an image is transformed via a multi- 
resolution decomposition process. In the proposed architecture, transforms such as 
KL, wavelet, wavelet package, lifting scheme, etc. can be placed in the transformation 
module. These transforms produce a set of decomposition coefficients {Cij} at 
different resolution levels and in different spatial orientations. 

10 The RICS architecture also supports DCT or windowed Fourier transform as a 
transformation technique. This is mainly for the transcodability with JPEG. It should 
be noted that the Fourier based transforms have been studied for more than a century; 
its theory is relatively complete and its mathematical and physical properties are weil 
understood. Particularly, its translation, scaling, and rotation properties may be very 

15 useful for content based retrieval computations. On the other hand, wavelet 
transforms are relatively new, and many of its properties require further 
investigations. As a result, the support of DCT may have an impact beyond the sole 
backward compatibility consideration. 

RICS also allows the NULL transformation (that is, no transformation is applied at 
20 all). In this case, an identity transformation is applied to the raw image data as the 
transformation step. The NULL transformation is useful in several instances. For 
example, it is usually not beneficial to apply DCT or wavelet transforms to bi-level 
images (text) for compression purpose. Another example is the residual images in 
video coding. The information in a residual image is the difference between video 
25 blocks and has a high frequency spectral content already. It is highly questionable 
whether it is beneficial at all to apply another mathematical decomposition (DCT or 
wavelet) to this type of data for the purpose of compact coding. 
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Region Definition 

The function of this module is to partition the coefficients produced by the 
transformation module into a number of spatial regions. The RJCS supports three 
region schemes. 

5 1. Automatic partition based on the preordering of the coefficients. 

2. Partition based on user defined ROI or object related shapes. 

3. Partition based on tiling. 

The first partition scheme is also referred to as the region hierarchy formation process. 
Consider the transformation coefficients as a set. The region hierarchy formation 
10 process partitions the set into a number of hierarchical ly disjoint subsets according to 
certain definitions of significance. In providing a very general partitioning technique 
captured in a very general region based control architecture, RJCS can perform highly 
flexible progressive transmission modes of operation that depend on data priorities set 
up for the code stream. 

[5 Scheme 2 deals with user specified ROI, typically primitive geometrical shapes, such 
as rectangles or circles, as well as object related shapes. Object related shapes could 
come from a variety of sources, such as user input, motion analysis in video 
compression, etc. 

Scheme 3 is a simple partition and requires minimal for shape coding. This scheme is 
0 essential in the JPEG mode. Some wavelet based compression techniques utilize this 
scheme to explore coding efficiency. This scheme offers very little support for 
content based accessibility. 

Hierarchically disjoint regions can be used in combination with user defined ROI in 
still image processing and objects in video processing. However providing a fair user 

5 partition for detail information is difficult in still image compression. But automatic 
partitioning or preordering techniques can be performed to control user selections in 
both arbitrary and block based modes of operation. The research presented in this 
document introduces a new multi-level control architecture for advanced multi-level 
image processing. The ability to perform a host of partitioning and preordering 

• techniques in both normal and region based modes of operation is encompassed in 
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this architecture. Both arbitrary and automatic region formation schemes are handled 
in the same manner at a high level. 

Region Shape Coding 

Three types of region shape are supported in RICS: tiling, primitive geometric shapes, 
5 and arbitrary shapes. 

1. Tiling requires only a small set of parameters to describe the configuration 
especially in sequence based processing modes. However the sequences can be 
organized and presented in many ways in packing them into the final code stream. 

2. Primitive geometrical shapes can also be coded efficiently. For example, a 
10 rectangle can be defined by four integers (jtnun^minjand (atma^ymax), or 

(jCmia, ytaiaj and (width, height) etc. 

3. Arbitrary shapes are more difficult and costly to encode. Partitions produced 
automatically by image analysis algorithms may contain many small regions. 
Specifically, the auto-region detection routines presented here produces 

15 hierarchically organized data partitions. This presents a highly challenging 
problem for shape coding. RICS provides two practical solutions to this problem. 
One is a quad tree based hierarchical region description. This mode of operation 
follows bit level ordering at different resolution levels (see embedded quad tree 
wavelet (EQW) in Ch. 4.) The other is a DCT coded multi-level region channel 

20 definition followed by preordering the partition (given the restraints of the code 

stream in terms of overhead). Though the coding efficiency of the second solution 
is currently slightly poorer than the first, it does contain a number of attractive 
features: 

• It provides a single representation for multi-level bitmaps. 

25 • From this single representation, region masks can be reproduced at arbitrary 
resolution levels, which is useful for subband coding. 

• It generates curved shapes, which potentially (e.g. in case of large number of 
complex curved regions) could outperform quad tree based region 
descriptions. 
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• It has several useful properties, such as translation and rotation properties, 
which the quad tree based description does not offer. In fact, a quad tree 
representation will change dramatically if an object is slightly translated in the 
image, making this type of representation not suitable for content based 
5 retrieval applications. However, the main advantage of the quad tree 

representation is its simplicity and most noticeably in block based modes. 

The region shape code is included in the code stream when the region definition is 
determined at the encoder. In the case of decoder specified regions, the region shape 
coding has a whole new meaning. 

10 Intra-Region Coding 

The function of the intra-region coding is to arrange the transformation data in an 
arbitrary shaped region into a one dimensional code stream. Regardless of the region 
definition scheme or the region coding technique, this streaming process requires an 
intermediate state where a control architecture can be designed to tailor the region 
15 channels whether dealing with a bitmap mask, an auto-detection routine or any other 
of numerous classification techniques. At the decoding end, the inverse routine 
generates the same mask to unpack the values from the one dimensional code stream 
and place them back into the correct position in each region. 

Intra-region coding is completed in block coders. Different block coders can be used 
20 to produce the ID code stream. For JPEG mode, the zigzag scanning/quantization 
routine is called to pack an 8x8 DCT block. For wavelet based coding, both explicit 
quantization and implicit quantization schemes can be used. In particular, embedded 
zero tree and embedded quad tree can be used as implicit quantization schemes. 
Furthermore most implicit quantization schemes are implicitly decodable. In dealing 
25 with bi-level images, a JBIG routine can be called a block coder. This effect can be 
staged by calling an implicit quantization scheme using one bit plane. Alternatively, 
efficient JBIG routines can be called block coders in an embedded coding scheme at 
each of the multiple bit planes. 

The Multiplexer 

30 The function of the multiplexer is to assemble the code blocks derived from different 

subbands and different regions in proper orders into a single code stream. Due to the 
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richness of region definition and block coding schemes, there are plenty of ways to 
pack the code blocks. Different ways to merging the data lead to different 
transmission priorities. For all transmission ordering modes, the final code stream can 
be transmitted progressively, and can be truncated at any desired place. 

The notion of using a multiplexer has been adopted in some standardized processes 
such as MPEG. In contrast, the use of a multiplexer in the design of a still image 
coding system is rare. The region based coding strategy opens the opportunity to 
systematically explore the syntactic and semantic richness in code stream ordering 
and transmission medium. With the image segmentation techniques improving in the 
future, region shapes will become more accurate in tracking objects in the image. In 
that case, the multiplexer will not only work as a syntactic composer, but also impose 
semantic meanings to the code stream. 

Highlight of Features 

The RICS is a true open architecture. It supports not only the algorithms DAC has 
developed, but also can accommodate most existing well known compression 
algorithms. Users may include their own functions that are appropriate to their 
application in a number of modules such as transformation, region definition, region 
shape coding, and intra-region coding all under the MUX control architecture. It is 
also implicitly flexible to new technological advances. 

• DAC's current implementation offers superior low bit rate performance that is 
competitive with best existing compression techniques. 

• The richness in region definition in the RICS allows great accessibility, thus 
providing a solid foundation for content based image applications. 

• It covers compression for both continuous tone and bi-level images in a single 
unified architecture. 

• It provides lossy and lossless compression in a single, natural, code stream in the 
course of progressive decoding. 

• It provides a variety of progressive transmission modes that allow images to be 
reconstructed with increasing pixel accuracy or spatial resolution using region 
priority modes for user specified ROI or system defined significant areas. 
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• It supports both fixed rate and fixed size modes. 

• It supports very flexible random access to and processing for regions with 
arbitrary shape. 

• It provides a graceful backward compatibility (or transcodability) with JPEG. 

5 • The generic region definition in RJCS is a very suitable interface for object-based 
coding schemes currently under development by MPEG-4. (In fact, as the 
arbitrary region shape begins to fit dynamic objects with more accuracy, motion 
estimation is more definite, and consequently allows for more efficient error 
compensation using region-based coding.) 

10 • Our general region shape definition provides a solid basis for object based 
composition and object based information embedding. Multiple objects with 
arbitrary shape are accepted. 

• Our bit stream is robust to bit errors. Each unit structure used by the MUX is 
independently decodabie. 

15 • Incorporating standardized encryption techniques with the RICS system is 
straightforward. 

Transforms 

In order to support multiple application needs, provide transcodability, and 
20 accommodate future growth, the RICS architecture is designed to support three 
categories of transformation: the DCT, wavelet transforms, and a special NULL 
transform. 

DCT 

The RICS architecture supports the DCT transforms as defined in the baseline JPEG. 

25 Wavelet Transform 

The RICS architecture supports various kinds of subband decomposition schemes, 
including the three schemes in Figure 3. 
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The Null Transform 

In dealing with the compound documents with mixed contents, it is sometimes 
required to encode some areas of the image without any transform. In particular, 
regions with bi-level pixels usually do not need to be transformed for coding purpose. 
5 In coding these regions, the transform stage is bypassed; the effect is simulated by a 
NULL transform stage in the process. 

The present version of RICS uses totally 34 types of wavelet kernels as given in the 
table in Figure 4. In our experiments, the biorthogonal filters (Bior9-15 of the table) 

10 seemed to give the best compression. Both low pass filtering and high pass filtering 
are done using convolution. After filtering, the wavelet coefficients are down 
sampled by 2. This process is repeated using the low pass part until the desired 
decomposition level is reached. In inverse wavelet transform, quadtrature mirror 
filters (QMF) for low and high pass are used. Then the coefficients are up-sampled 

15 by 2. 

Color Transform 

In addition to YUV format, the RICS system uses the Karhunen-Loeve Transform for 
color transformation. Following the notation of statistics we term this process as the 
color standardization. The KL color transform requires more computation than YUV 
20 transform. Figure 5 shows the PSNR comparison of the two color transforms. More 
test results are also available from DAC. 

Region Definition and Shape Coding 

The notion of region processing plays a fundamental role in the operation of RICS. 
25 The choice of region based coding is motivated by many application needs such as 
content based retrieval, interactive multimedia application, graphics object and image 
composition, and coding of dynamic object in video compression. 

There are two fundamental issues in a region based coding strategy: defining regions 
and coding the regions. In the RICS system, region coding is divided into the shape 
30 coding and the intra-region coding (the content coding). Section 3, describes the 
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various schemes for region definition and shape coding. The intra-region coding is 
discussed below. 

From the viewpoint of JPEG 2000, the process of region definition does not have to 
be standardized. However, schemes for forming regions that the JPEG 2000 will need 
5 to support should be defined clearly. The R1CS supports three types of region 
definition. 

• Tile based organization 

• primitive geometric representations 

• arbitrary shapes 

10 Region definition is an optional step: an image can be coded without specifying any 
region (the non-region mode). In this case, the entire image area is considered as a 
single region. 

Tiling: Definitions and Shape Coding 

Tiling is perhaps the simplest region definition scheme. In this scheme, the entire 
15 image area is divided into a number of rectangular blocks. In particular, 8 by 8 tiles 
are used in the JPEG coding mode. Most tiling approaches cannot cover the natural 
shapes, region trends or partial objects in a picture. 

For coding a tiling scheme, a small set of global parameters will be sufficient. The 
primary parameter is the size of tiling block and the technology is developed around 
20 this theme. In case of subband decomposition, tiling block size may vary from one 
subband to another. However, the RICS architecture has a sound operational 
foundation. It can be operated in both tile based or arbitrary processing modes of 
optional arbitrary region formation schemes. 

Primitive Geometrical Shapes 

25 Primitive geometrical shapes are ideal for supporting user interaction; i.e. user 

specified ROI, and for processing compound documents where the text areas can be 
well covered with one or more rectangular boxes. Geometric shapes currently 
supported by RICS are listed in the Table in Figure 6. 
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Arbitrary Regions and a Generic Binary Partition Hierarchy 

Let C 0 be the set of image primitives. Given an ordering relationship >■ , generic 
binaries partition of C 0 into C l0 and C n is defined by the following conditions. 

(1) C 10 uC„ = C 0 , 

(2) C )0 oC u = <J>,and (Eq.m.l) 

(3) for any a e C l0 and b e C n , a>b holds. 



Recursively, each of the new generated subsets can be further partitioned into two 
10 subsets, resulting in a hierarchical structure to represent the partition (see Figure 7) 

A useful ordering relation popular in the compression community is "more significant 
than": a > b whereby a is more significant than b. The definition of significance can 
be very general. One definition may be used to create an ordering, or several 

15 definitions are combined together to generate an ordering. This can be illustrated by 
the following example shown in Figure 8. Suppose the region masks shown in the 
leftmost image are generated by considering the magnitude of wavelet transform 
coefficients as the measure of significance. The region mask in the middle image is 
an object shape generated by another definition of importance that, for example, can 

20 be the intensive dynamic region in a video frame. The rightmost image shows the 
intersection of the two region masks, which is a new region partitioned according to a 
multiple definition of significance. 

As an example, if the significance is measured by magnitude of wavelet transform 
25 coefficients, then the ordering relation is simply the numeric ">" relation. To create a 
binary partition, a set of thresholds is sufficient. The first threshold, say T 0 , separate 
the initial set C 0 into two subsets, C 10 and C n . Subsequent threshold values can be 
specified to recursively partition the new subsets. 

The above partition scheme produces accurately a number of disjoint subsets which 
30 satisfies the strict ordering relationship of >~ . However, it usually produces many 
small, scattered regions. Consequently, the representation of region shapes for these 
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subsets can be potentially expensive. From the representation point of view, fewer 
numbers of smoothly shaped regions are desirable. In the following, a less strict 
partition scheme is defined, which takes the above partition scheme as a special case. 

Instead of requiring the condition a>b to be held for all aeC lQy &€C„, it is 
replaced by a looser condition 

-.AeC,, a, for any aeC 0 . (Eq.m.2) 

Intuitively speaking, this definition defines a partition in which no element in C, 

proceeds any element of C 0 . In other words, some elements belonging to C, (by 

Eq.m.l) may now be placed in C 0 , but, no element belonging to C 0 C, (by Eq.m.l) 

can be placed in C, (Eq.HI.2). In doing so, the subset C 0 will be enlarged, and the 

region shape of C 0 will be smoother. At the same time, the enlargement of C 0 is 

controlled to such an extent that the loss of classification accuracy is limited to a 
reasonably low level. 

Another scheme, which is symmetric to the Eq.III.2, can be defined similarly for the 
following condition. 

-^3gC 0 , a<b> forany b e C, . (Eq.m.3) 

In this case, we ensure that no element in C 0 is proceeded by any element in C, . 

It should be pointed out that there is a tradeoff between the enlargement of C 0 and the 
space requirement for the region shape representation. It is possible (and indeed quite 
often) that there are some isolated scattered points that belong toC 0 . But, in order to 

exclude those isolated points from C,, we have to sacrifice many elements from C, . 
This will result in a very large C 0 , a degraded classification. An approximation to the 
scheme of Eq.m.2 is introduced to rectify this problem. A small number of elements 
in C 0 are allowed to be "misclassified" into C, . Those misclassified elements can be 
separated later in the intra-region coding stage that generally follows natural 
processing orders available for further classification of the partitioned elements. 
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When such region hierarchy formation schemes are applied to image primitive planes, 
they serve as preordering processes that set up a partial ordering among the 
primitives. Then, the problem of coding this ordering becomes a geometry problem 
for coding the shapes of the partitioned regions. Transformations such as wavelet are 
5 known to use a joint spatial/frequency domain methodology. The region hierarchy 
formation scheme mentioned above has an intuitive geometric explanation for this 
class. Transformations such as DCT and Fourier transform are purely frequency 
domain methods. It is interesting to note that when the above partitioning scheme is 
applied to a 2D DCT transformation plane, the generated regions are roughly 
10 circularly shaped bands. The popular zigzag sequence used in JPEG turns out to be a 
simple and reasonable approximation of these bands. Therefore, the region based 
coding strategy accommodates naturally both spatial/frequency domains and pure 
frequency domain transformations. 

Automatic Region Formation in the Wavelet Transform Domain 

15 DAC has developed an automatic region formation scheme that is used to categorize 
wavelet coefficients in the transform domain according to magnitude. The procedure 
will be outlined in the following sections. Also included are many of the design 
details used to develop the first version of the general region classification scheme. 
The advantages and disadvantages of the technique as well as performance and 

20 overhead issues will be discussed. Finally conclusions will be drawn to summarize 
the results. 

Detecting the Regions 

The wavelet transform is used to decompose the original image. information into a 
multi-resolution hierarchy of data that is more suitable for compression. The result of 

25 completing one pass of the 2D wavelet transform is one low pass part (LLi) and three 
high pass parts (HLi, LHi and HHi.) The transformation procedure is repeated using 
the low pass LL part as the starting point for each succeeding pass. The LL part is an 
approximation of the original image at a lower resolution. At each resolution level 
the HL parts carry the vertical, the LH parts carry the horizontal and the HH parts 

30 carry the diagonal edge information. The level of detail information contained in any 
particular orientation is specific to the local response generated in the choice of 

19 

SUBSTITUTE SHEET (RULE 26) 



BNSDOCID: <WO nrun*7iAi t*> 



WO 00/49571 



PCT/CA00/00134 



wavelet kernel. The original image is transformed into a unique band pass response 
hierarchy of detail information. In any compression process, it is the highest energy 
coefficients that are considered first for making a contribution to the reconstructed 
image. It is this transformation property that is used to partition the coefficients into 
5 regions of interest corresponding to high energy locations in the original image space. 

The magnitude of the coefficients in a particular subband is related to the response 
that the original image information has to the transform kernel. Sharp edges in the 
original image (or a lower resolution LL part) are indicated by an increased kernel 
response in terms of coefficient magnitude, in the general vicinity of a specified area 
10 of concern. Other image changes that are not as abrupt will translate into a gradual 
response that is less significant. In those areas where little or no change occurs, the 
wavelet transform will indicate little or no response. 

Level 1 Kernel Response Threshold Images 

15 DAC's automatic region formation scheme makes use of coefficient magnitude 
information to categorize the data. The starting point for the procedure is at the 
highest resolution level of the wavelet transform hierarchy looking at the three detail 
data sets (HLi, LHj, HHj.) A 2 bit representation of each detail orientation for a 256 
by 256 Lena image is given in Figure 9. These images are obtained by using 

20 threshold values to categorize the coefficients by magnitude. The wavelet kernel used 
in this case is the standard 9-7 filter set implemented in a lifting scheme. 

A decaying histogram procedure is used to determine the threshold values in each 
case. In this particular analysis the coefficients are partitioned into regions according 
to decreasing levels of magnitude; 10% of coefficients are partitioned into region 1, 
25 15% into region 2, 25% into region 3 and 50% into region 4. Note that the largest 
coefficients appear darker in the images while the smallest appear lighter. The 
threshold values are calculated automatically for each orientation. The values used to 
generate the partition in each case are given in Eq.m.4 (assume Q is the magnitude of 
the coefficient under consideration.) 

30 
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HLj: 0 < Q < 2, 2 < Q < 4, 4 < Q < 8 and Q > 8 
LHj: 0<Ci<3,3^Ci<5,5<C ; <14andCi>14 (Eq.IH.4) 
HHi: 0 < C, < 2, 2 < Q < 3, 3 < Ci < 5 and Q ^ 5 

The set of threshold images appearing in Figure 9 is motivational in determining a 
region formation scheme based on this approach. It is apparent that wavelet 
coefficients can be classified in this manner to form a mask. However, the large 
overhead required to code each individual mask must be solved in order to make this 
type of classification scheme viable for implementation. 

Formation of the Raw Common Mask 

The first step taken in the development of a technique designed to reduce the mask 
overhead is to consider a common mask approach that can be used to capture the most 
important coefficients between all three orientations at the lowest resolution level. 
Given that the data range is different in each orientation, the data is normalized to the 
largest range in an absolute value sense. This step is taken in order to compensate for 
the range differences that exist between the data sets, and to ensure that corresponding 
pixels from each orientation can make an equal contribution in the formation of the 
common mask. This step is illustrated in Eq.ITJ.5. 

HL, = NewRangefHL,) - ScaleRange(MaxRange(LH|, HL, , HH f )) 
LH', = NcwRange(LH, ) ■= ScaleRange(MaxRange(LH, , HLj , HH, )) (Eq.m.5) 
HH | = NcwRangc(HH|) = ScateRange(MaxRange(LH|, HL, , HH,)) 

Once the data ranges for the smallest two sets have been scaled to that of the largest, 
the common mask values can be determined. The next step is to generate a new 
normalized data set by taking the maximum value between scaled orientations at each 
pixel location. Let C (i) be the new coefficient obtained in this procedure and let 
C hli , C lhi and C hhi be the individual scaled values from each orientation. The 
new data set is formed using the step illustrated in Eq.m.6. 

C'(i) = MAX(ABS(C'hli(0X ABS(C'hli(D), ABS(C*HLi(i))) (Eq.m.6) 
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The final step in forming the common mask is to threshold the new coefficients based 
on the histogram decay and the percentage of the total data amount to encapsulate in 
each region category. The same values mentioned earlier are used in this case (10%, 
15%, 25% and 50%.) The raw common mask categorization image appears in Figure 
5 10. The threshold values used to generate the mask are given in Eq.IH.7. 

HV 0 < Q < 5, 5 < C; < 11, 1 1 < Q <26 and Q > 26 (Eq.ID.7) 

10 

Note the level of detail contained in this image. High frequency edge information 
representing the most important coefficients appears in the darkest areas. Low 
frequency changes representing areas of the image that are relatively uniform appear 
in the lightest areas. The combined kernel response from each orientation follows the 
15 high energy areas that exist in the original image. 

Common Mask Conditioning Using the DCT Method 

Generally, the raw size overhead of the common mask obtained in the previous 
section is still too large to be included in the final bit stream for most image 
20 processing needs (i.e. an overhead of 2 Bpp for this particular 4 region mask). This 
section outlines the next step in the automatic region formation scheme. The DCT 
transform is used to reduce the overhead size to an acceptable amount. 

The DCT is not applied directly to the common mask data in its mult-valued form. 
Rather it starts by considering each region contained in the common mask as an 

25 binary data set such that the sum of the individual data sets is equal to the multi- 
valued mask in functional form. The DCT transform applies the transform of the sum 
of functions in the same manner as taking the transform of each individual function 
and summing the results. Furthermore there are unique spectra in each case that 
should be considered separately for a fully integrated analysis. The individual region 

30 masks obtained by decomposing the raw common mask for the Lena image appear in 
Figure 13. Notice that although there are only three images, the fourth region channel 
(the background) is implicit. 

The 2D fast DCT algorithm is used to transform each binary data set into the 

frequency domain for spectral analysis. The spectra for the upper three regions 
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appear in Figure 12. Please note that a log scaling of the coefficients is used in the 
images to exaggerate the appearance for display purposes. Notice how the low energy 
content is dispersed through the spectrum in each case. Even at such low energy 
levels, the energy is localized in some areas. 

It is apparent from the spectral results of Figure 12 that the magnitude of the DCT 
coefficients decreases rapidly in moving down each spectrum. The interaction of the 
content retained for each partition is used to guide the classification procedure. The 
interrelationships that exist in each binary spectrum must be investigated further. 

A plot of the overall summed spectra in short integer format appears in Figure 13. 
The conventional zigzag ordering is used to re-group the coefficients before 
quantization. Note that only the first half of the data in reflected in the plot. The 
outer 8K are omitted since they do not change much from the DC level. 

The first consideration observed in the plot is that the coefficient magnitude 
deteriorates at an alarming rate over a relatively small number of coefficients in the 
first portion of the spectrum. The second observation is that most of the spectral 
energy is concentrated in the first 12% of the data. Together these observations 
suggest that implementing a low pass spectral filtering stage, followed by careful 
quantization will reduce the amount of data that must be retained to capture the region 
formation concepts at the lowest level ( in this common mask partitioning scheme). 
Note the apparently large dc offset. Part of this level is introduced by scaling the 
original wavelet data sets to a unified range. The rest of the offset is due to the DCT 
transform. 

Low Pass Filtering the Common Mask Spectrum 

The total mask spectrum must be filtered carefully to reduce the excess content that 
has little contribution to the region trends that exist in the retained coefficients. The 
main concern is to determine how to quantify the filter size such that a dynamic 
implementation may be obtained that will work accommodate any arbitrary image 
sizes. 

The first step taken towards finding a dynamic solution to this problem comes from 
experimentation. In looking at the result of using zigzag ordering, coefficient 
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magnitude and significance deteriorate at some exponential rate. Experimental 
evidence suggests that as the size of the original image increases, the number of 
coefficients that must be retained to construct a similar quality mask increase on a 
logarithmic scale. This is an important observation since doubling the original image 
5 dimensions translates into a log 2 increase, in the number of retained spectral 
coefficients. 

Assume that P is a general log base to begin our discussion. If the filter size for an N 
by N common mask spectrum has been determined experimentally as 4>n, then what is 
the size of the filter for a 2N by 2N mask spectrum? If the unknown filter size as <|>2n, 
10 then the following simple logarithmic relationship can be used to compute the new 
filter size. 

i c 128 

15 -Log* (♦«/*«) 

256 

(Eq.m.8) 

20 



A number of experiments were conducted to determine a reasonable filter size that 
can be used as a base for filtering common mask spectra. It has been determined that 
for masks of size 128 by 128, a filter size of 40 to 45 diagonal rows yields reasonable 
25 results. The table in Figure 14 give the filter sizes for raw mask sizes using 1.375 for 
p. The logarithmic base P can be used for final calibration in the current design 
implementation. 

Notice the filter size specified for a common mask spectrum of 32 by 32 is given as 
32 diagonal rows. This value is selected to calibrate the filter sizes for the larger data 
30 sets. The results indicate the filter size required for a first cut of the common mask 
coefficients does not have to be exact since the distribution of the spectral content 
may change slightly from one image to the next. However, it must be large enough to 
capture the majority of the most important spectral content for each region partition. 
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Spectral Quantization Techniques for Filtered Coefficients 

DAC has developed two techniques for quantizing the filtered mask spectrum. The 
first approach is basically a modified uniform quantization procedure that is used to 
verify the concepts. It proved useful in confirming the algorithms and developing the 
5 concepts, but it lacks robustness and flexibility. The second technique is a general 
approach that is much more robust and elegant. It is dynamic since it tracks the 
magnitude of the spectral content in determining how to quantize the coefficients. 

A First Quantization Approach 

The first approach combines the filter dimension determination into the quantization 
10 procedure. A base mask size of 128 by 128 is use as the starting point in this 
approach. Assuming that a logarithmic relationship exists for coefficient importance 
along the diagonal order, a quantization technique can be developed to exploit the 
relationship. 

The main premise is that there are bands of spectral coefficients that can be used to 
15 guide the quantization procedure. The importance of the coefficients in each band 
decreases along the spectrum. This concept is illustrated in Figure 15. 

The number of diagonal zigzag rows to include in each band is a function of the filter 
size and the original common mask size as well as the spectral content. As a first 
approximation, a table look up technique is used to determine the number of rows to 
20 use in each band. The base band sizes determined experimentally to give reasonable 
partition and reconstruction results for a 128 by 128 mask are given in the table in 
Figure 16. 

Notice that the number of diagonal rows doubles in each step. The number of bits 
25 selected to quantize each band decreases from one band to the next in the following 
fashion. 

• Band 1:16 bits 

• Band 2: 10 bits 

• Band 3: 8 bits 
30 • Band 4: 8 bits 
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A series of band sizes were found experimentally of mask dimensions by powers of 2 
increments. The results are in the table of Figure 17. These values are used to 
partition the spectral content for quantization. This information together with the 
number of bits used in each band and the filter size measurement determines the 
5 common mask overhead of an image. 

The mask overhead based for this quantization scheme is tabulated in the table in 
Figure 18 for the corresponding image sizes. There is an additional overhead header 
included in this result for mask reconstruction on the decoder side. Note that the 
mask overhead is the same size for both grayscale and color images since it is 
10 generated from the intensity information. However, the percentage overhead in the 
color case is reduced by a factor of 3 since it can be distributed over 3 channels. 

After applying the inverse DCT on the quantized coefficients a low pass 
approximation of the original common mask is obtained. It is this result that is used 
to guide the region channels in classifying and processing the wavelet coefficients. 
15 The result indicated in Figure 19 is obtained by applying this technique to the first 
wavelet transformed level for a 256 by 256 Lena image. From Table 17, the overhead 
for this mask is 1021 bytes. 

Notice the high energy changes have been eliminated by low pass filtering (especially 
the changes that occur over small portions of the image). However, the mask does 
20 succeed in capturing most of the essential parts of the original image. This mask is 
common to all orientation at this level and used to partition the coefficients into 
region categories. 

A Second Quantization Approach 

There are some apparent problems with the underlying procedures used for 
25 quantization in the first approach. The first of these is that the coefficient band 
classification is fairly rigid and not that flexible. The filtering and quantization stages 
are tied together and fully independent. Affecting one parameter has consequences to 
the other. It is true that a logarithmic relationship exists in the spectrum that can be 
exploited to determine a reasonable filter size, but the quantization step must be 
30 tailored from the filtering stage. 
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DAC is currently implementing another quantization scheme for mask filtering that is 
much more dynamic and flexible. Quantization is no longer tied to the filtering stage 
and the coefficient banding technique of the previous approach. The current 
technique follows the actual content of the spectrum in forming quantization 
5 categories. In this way, coefficients are grouped into categories according to the 
energy gradient of the spectra! distribution. The concept is illustrated in Figure 20. 

The rate of spectral decent together with position information within the spectrum can 
be used to quantize the number of bits used to categorize the data. This process can 
be controlled precisely. The procedure can be calibrated as required by studying the 
10 effects of changing image dimension in terms of an optimal quantization change. 



The previous quantization technique can be used to estimate the mask overhead for 
the new scheme proposed here. Consider the same base conditions presented for the 
15 banding approach applied to the 256 by 256 Lena image. The spectral filter size in 
this case is 44 diagonal rows. If 4 quantization intervals are used at 16, 12, 8 and 4 
bits each the mask overhead can be estimated. Let N; be the number of coefficients in 
each interval, bi the number of bits used to quantize this interval, and Om be the total 
mask overhead. A rough estimate of the new packed size is calculated in Eq.m.9. 

3 

Qm =£ H.b, 

25 iB ° (Eq.III.9) 

Om = 6x 16 + 39x 12+ 186 x 8 + 804 x4 

Ou = 5268 bits (-659 bytes) 



Even though the actual overhead has not yet been determined, from the insight gained 
in using the first quantization approach a saving of 30% is not unreasonable. This 
promises to be a substantial saving in the mask overhead. .There are ways to utilize 
35 saved overhead. The first is to reduce the amount of mask information in the code 
stream. This approach will retain the same mask quality leaving extra space for 
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refinement information in the code stream. The second way to utilize the saved 
overhead is to retain a better quality mask in keeping the overhead size constant. 

The Common Mask and Multi-resolution Classification 

A translation technique is required to apply the region mask at other resolution levels. 
The simplest approach is to look at the significance of the mask coefficients in a small 
block to determine an appropriate value for the next resolution level. This process is 
repeated until a mask for each resolution is obtained. The most logical approach is to 
take the largest coefficient through to the next level. 

Mvfl>i W = , M^tas+ui » Mv<L)a.3fi , AfyLja+uKi) (EqUL 10) 



Some heuristic approaches have been investigated for resolution translation. One 
approach is to make decisions based on the sum of the individual mask coefficients. 
The upper level masks are affected by either enlarging or shrinking the encompassed 
coverage regions. However, there is currently no conclusive evidence confirming the 
validity of this approach and more experimentation is required. 

It is worth mentioning at this point, that DAC has begun investigating the wavelet 
kernel ROI mask formation technique of verification model (VM) 3.0 as a method of 
resolution translation for other transformation levels. Preliminary investigations 
indicate this technique introduces a gradient bias for regions of higher classification 
level in the translation. A combined hybrid approach may have some advantages. 

The rest of the common mask hierarchy obtained by using the simple 4 to 1 resolution 
translation techniques and observing the most important region as key is given in 
Figure 21. The complete set of region masks is used to classify regions of importance 
in the wavelet transform domain according to coefficient magnitude. 

Misclassification and the Common Region Mask 

It should be noted that while it is true that misclassification occurs in this technique, 
there is a tradeoff between code stream overhead and region channel accuracy. The 
original raw common mask size is 32 kBits for a 4 region mask. The basic uniform 
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quantization scheme compresses this amount to 8 kBits. The modified scheme 
suggests that mask compression of 5 or 6 to 1 can be achieved with a tradeoff between 
mask accuracy and code stream overhead. 

The low pass filtering and spectral quantization steps can be tailored to deliver similar 
5 quality masks for most images in a dynamic implementation. In addition the partition 
can be studied in regards to how the original grouping is affected by the filtering step. 
For the Lena image example the original partition is 10/15/25/50%. After filtering the 
coverage ratios used for bit budget calculation and the code stream distribution (see 
multiplexer in Ch.V.) changed to approximately 9/31/33/27%. There may be some 
10 advantages in adjusting the original partition such that the data coverage ratios are 
controllable in the final stages. 

The question of miscoverage must be addressed in all lossy region formation 
techniques. It is the subsequent modules that are affected by the misclassified 
information. Thus the interaction between the DCT region formation module and the 
15 subsequent sorting and packing modules must be understood in any attempt to obtain 
the best region channel results for this classification scheme. 

Handling Misclassifi c ation in a Region Formation Scheme 

Subsequent stages in the compression algorithm are already tailored to process 
20 information in a descending level of importance. Most wavelet implicit quantization 
procedures process the coefficients in bit plane order of importance. What this means 
in terms of any region formation scheme is that the information encompassed in each 
region is further partitioned by bit level processing. In the common mask approach, 
further processing causes the most important misclassified coefficients to filter to the 
25 top at a faster rate than misclassified coefficients of lower importance. Effectively bit 
level processing can be considered as a second partitioning stage that deals with 
misclassification. 

There is a tradeoff between the accuracy of the original region partition and the 
additional bit level processing overhead placed on subsequent sorting stages. It is the 
30 final processing stages that implicitly accomplish the final coefficient partitioning. 
DAC is currently using two bit plane partitioning core technologies for both grayscale 
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and color images, and binary partitioning for bi-leve! images is now under 
implementation. 

Bit Plane Sorting for Common Mask Miscoveraee 

The concept of bit plane sorting is a well know technique used to organize priority in 
a distributed list of data. ID sorting a list of information generates a map specific to 
that list in an optimal fashion. The largest values are mapped first as the list 
transposition routine progresses. The order of decent is the standard order used to 
classify information if considered from the normal processing standpoint where no 
regions are involved at all. As a result, most of the misclassified information is 
considered for further classification and are thus biased to be included first in the final 
code stream order. The only difference is that now there is a region description 
overhead as well as the final mapping overhead of sorting the region of interest 
partitions in a hierarchical fashion. 

Currently DAC has not documented the exact overhead introduced by region 
processing in comparison to normal processing modes that do not use regions. But 
the final results obtained by using the common mask approach are encouraging to say 
the least. At this level the underlying routines do not care where the information to be 
sorted originated. The ordering technique will run to completion whether region 
channels are involved or not The original misclassified coefficients generated in the 
first cut will filter through to the final ordering set by the sorting stage. There is an 
important tradeoff here between the sorting cost and the region overhead that must be 
considered in the final design implementation. 

Bit Plane Ordering for Common Mask Miscoverage 

DAC has developed a quad tree ordering technique (see EQW Ch.4) to track 
coefficient importance in both normal and region based processing modes of 
operation. The normal quad tree approach is to track the leading one position through 
each block under consideration in a recursive fashion. A map is generated in the 
process for each entry based on importance such that decoding follows the same 
order. At this basic level the quad tree order is an efficient method of processing the 
multi-resolution decomposition image information in an effective manner. 
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The tracking or positioning overhead introduced by mapping the data is generally 
smaller than in the ID sorting case. Retaining vertical and horizontal positioning 
information is highly advantageous (in most image processing applications) since the 
underlying technology is generally consistent in this medium. The ID sorting 
5 approach sacrifices the natural vertical correlation that may exit (although it can be 
partially recovered by bit level entropy coding in the sorting routine.) However each 
of the two sorting techniques can be used from an operational standpoint to produce a 
similar net effect. One may produce slightly better results than the next, but the 
control architecture is consistent. Each final ordering routine can be tuned to meet 
10 specific processing needs. The ID sorting approach may be a useful candidate for bi- 
level image processing, and DAC is currently investigating this possibility. 

The quad tree approach has currently been modified to operate in arbitrary region 
processing modes. The ID sorting routine tracks the importance in a list of 
information specific to the block under consideration. That is the region partition is 
15 distributed in ID hierarchical lists of information. The original preordering of the 
coefficients is captured in the common mask procedure. Thus the decoder map 
already exists for the inverse ordering. 

Initially DAC's region processing quad tree begins by building an information map 
for each region channel to guide the classification. The core engine has been 
!0 modified to drops blocks not contributing to the final ordering in each region 
category. Currently the overhead comparisons for each sorting case have not been 
documented. The results will be available from DAC once the region based quad tree 
design implementation is finalized (currently under implementation at DAC.) 

In each case, the misclassified coefficients obtained from the original preorder mask 
5 partition are considered again for ordering in the sorting stages 1 that follow. 
Effectively, the most important misclassified coefficients still have the opportunity to 
make important contributions to the final code stream as well as improving the 
reconstruction quality measurements in the decoded image. The difference between 
this approach and most efficient classification approaches is that the original 
> processing order used of the efficient scheme is somewhat distributed through each 
region channels. However the misclassification in each region channel is forgiven to 
a certain extend the subsequent bit level sorting module. One of the prices that must 
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be paid for any automatic region classification scheme is that the optimal processing 
orders that exist in standard approaches must be considered in a different context. 
The original ordering concepts used for standard procedures must be extended to 
include other ordering techniques specifically tailored for operation in arbitrary region 
processing. 

General Region Processing Concerns 

There is one important hurdle that must be overcome in the design of any arbitrary 
region processing technology. There are times when rigid tile or block control is the 
optimal way to process image information. However, it is difficult to achieve 
arbitrary accuracy in a strict processing order. DAC's ID and 2D sorting techniques 
address this issue in design implementation. In our opinion arbitrary region 
processing channels can be defined in a still image environment. Careful study and 
analysis suggests that this is a viable approach not only for the DCT common mask 
approach but also for other arbitrary region formations techniques that have not yet 
been developed. Furthermore region classification may have interesting properties 
that can transferred to video processing environment where the region of interest 
concept and the object motion vectors can be considered together in a unified 
approach to region of interest processing. 

DAC's region technology in its original form was designed to include region 
processing capabilities from the start. AJ1 internal modules can be operated in both 
normal and region based modes of operation. The MUX architecture (introduced in 
Chapter V) is designed to meet operational requirements in both normal and region 
processing modes. It is possible to extend the notion of block based image processing 
to a level where a block of information is interpreted in an arbitrary way that depends 
only on the underlying core processing technology. In this way, blocks can be 
considered as individual processing units each fully capable of making a contribution 
to the final code stream. Furthermore control architecture can be developed to fully 
exploit the region processing capability. All region formation schemes, whether 
arbitrarily defined user regions of interest, automatic region formation schemes or any 
other specially designed region classification technique, can be processed in a 
common framework. The same overall processing architecture can be tailored to 
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operate in an optimal fashion for both normal and region processing modes of 
operation. 

Additional Processing Concerns 

One of the important concerns in the JPEG-2000 community regarding the 
5 development of a new compression standard is that it must be able to support ROI 
processing: There are a number of techniques that have been tested with some 
success. Some of these techniques are listed below. 

• Sequence based mode. When this mode of operation is used, each set of data 
encapsulated by a specific region is treated as a separate sequence. The sequences 

10 are coded independently. 

• Scaling based mode. In this mode of operation, the magnitude of each ROr 
coefficient is increased using a predetermined shift factor. If multiple ROIs are 
used with increasing levels of importance, multiple shift factors distinguish each 
ROI. The effect of this scaling technique is to force the ROI coefficients to be 

15 encoded first. 

• ROI from the middle. In this mode of operation, a concern of coding an ROI in 
the middle of the bit stream is addressed. This is useful for progressive 
transmission of images where the user is given the opportunity to focus on a 
specific ROI before the complete set of image data arrives at the decoder. This is 

20 important for medical image processing. 

• ROI without sending mask information. This mode of operation is a special case 
for scaling based mode. The coefficients for each region are scaled such that the 
magnitude of each ROI exceeds that of the ROI of less importance. The decoder 
knows which ROI the coefficients belong to based on the magnitude. 

25 The ROI coding technique that DAC has developed generally falls into the sequence 
based category. But internal modules have been tailored to operate in both block 
based and arbitrary block sized processing units. DAC is currently studying the 
different mask and region formation techniques to determine the effect each ordering 
and partitioning approach has on the overall organizational structure required for 
region channel processing. The current implementation includes a scalable ROI mode 
of operation implemented without actually shifting the ROI coefficients encompassed 
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by the classification. Data scaling is handled in the MUX control architecture. 
Internally, the scaled version is a special case that can be implemented in MUX 
control. 

Preliminary study has begun for the last of the four categories cited above. ROI from 
the middle implementations can be addressed by using resolution a progressive 
processing order. This type of ordering is supported in the MUX architecture. DAC 
is currently considering a design implementation for client side region of interest 
requests in a resolution progressive architecture for client-server region interaction. 

ROI without mask information makes sense in theory. However it places a huge tax 
on the subsequent sorting and classification stages. Generally speaking, the overhead 
cost introduced by additional sorting is comparable to the size of a bitmap mask that 
could have been used to classify the data initially. The shift introduced to each ROI 
must be processed in the sorting stage. Eventually sorting must traverse from top to 
bottom. Coefficient scaling under tight control has some definite benefits in many 
cases. However it would be highly inefficient in a general region processing 
architecture because of the high sorting costs as a result of traversing all region 
channels. In a region formation scheme of 4 or more regions, the total bit level 
difference between all ROI is too large to process entirely. On the other hand, if the 
resolution of the overall partition is reduced (less bit levels), less room is available to 
form an accurate region classification. There is some common ground that must be 
explored in order to determine the optimal processing mix. 

Intra-Region Coding 

The segregation of image primitives into regions provides a basis for accessing the 
image contents. Compact representation of image primitives is achieved via intra- 
region coding. 

Independent coding units (ICUs) are the building blocks for intra-region coding. In 
RICS, intra-region coding with ICUs is designed with a set of objectives. 

(1) Full data independence. From the notion of ICU, the encoding and decoding of a 
region is done without referring to the data of any other regions. 
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(2) Allowance for multiple coding schemes. Each ICU may be coded using different 
coding schemes. New coding schemes can be added to the system (modular 
openness.) 

(3) High coding efficiency. Depending on the characteristics of the primitives in a 
5 given ICU, one or more efficient coding schemes can be selected to produce a 

code block with high compactness. 

(4) JPEG and JBIG transcodability. It is preferred to have a single JPEG 2000 
coding platform that also accommodates JPEG and JBIG modes of coding. 

(5) Scalability of code stream. 

10 (6) Error resilience. The ICUs are natural blocks for bit error localization. 

(7) Parallelism. There is no data dependence between ICUs and so the encoding and 
decoding of all ICUs can be performed in parallel. 

Intra-region coding in RICS involves the following steps. 
(1) Choosing a coding scheme. 

15 (2) Determining ICU structure. 

(3) Coding. 

(4) Pre-packing. 

In the rest of this section the first three steps are described. 
Choosing a Coding Scheme 



20 



For coding an ICU, the current RICS design employs six categories of coding 
schemes. 



• Zigzag DCT Coefficients Packing (JPEG baseline scheme) 

• Embedded Quad tree (or similar schemes) 

• Embedded Zero tree (or similar schemes) 
25 • ID Progressive Sorting Schemes 

• Block Based Quantization 

• JBIG Coding Algorithms 
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There is no general restriction on what schemes have to be used for coding a given 
ICU. However, certain preferred embodiments may apply. For example, if DCT is 
used at the transform step, then the zigzag DCT coefficient packing might be 
preferred (but not restricted to). 

Choosing ICU Structures 

While being a logic concept, an ICU still has a geometric structure. The ICU 
structure is directly relevant to the coding scheme chosen. Currently, the RJCS 
recognizes three types of ICU structures (Figure 22). 

Type 1 ICU Structure: 8x8 Blocks 

This type of ICU is designed exclusively for use in JPEG mode, although it is not 
necessary that DCT transform coefficients be coded this way. 

If there is no region defined in the image and the JPEG mode is chosen, the entire 
image is paved with type-] ICUs. 

If a region R is to be coded using JPEG mode, the set of ICUs that covers R form a 
unique, minimal pavement for R t in the following procedure. 

Step J. Let j'min be the first row from the top that has at least one pixel in R, ym*x the 
last TO Wj X nun the first column from the left, and x m» the last column. 

Step 2. Let (xmin^min) and (xm^ymw) be respectively the top-left and the 

bottom-right corners of the bounding rectangle of J 1 ?. 

Step 3. Starting from the top left corner, pave the bounding box completely with 
type-1 ICUs, 

Step 4. Remove those ICUs that cover no pixels in R. 

After this procedure, the remaining ICUs form the minimal pavement of R (Figure 
22). 

Type 2 ICU Structure: Rectangles 

Type-2 ICUs are rectangles. Except for the JPEG and embedded zero tree schemes, 
any other intra-region coding scheme uses type-2 ICUs to pave and code a region. 
There is no general restriction on the dimensions of a type-2 ICU. It is usually 
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defined by the selected coding scheme. For example, both embedded quad tree and 
explicit block based quantization can use arbitrary sized rectangles, with some 
preferred embodiments (such as the 64x64 blocks used in EBCOT of VM 3.0 (A)). It 
should be noted that once a region is paved by type-2 ICUs, different coding schemes 
5 can be used for each ICU. 

For subband decompositions, no type-2 ]CU is allowed to cross subband borders. 
Essentially, this means that type-2 ICUs support only various types of intra-band 
coding methods. 

Tvpe-3 ICU Structure: Pyramids 

10 The third type of ICU is the pyramid structure, which is used by various inter-band 
coding methods, such as the embedded zero tree. In the current RICS design, there is 
one limitation on the region definition procedure for type-3 ICUs: a region must be 
specified in the top-down manner, from lower resolution to higher resolution in the 
decomposition pyramid. That is, for a given region definition R in LL, every element 

15 in R defines a set of type-3 ICUs (i.e. in three spatial orientations, respectively). 

Let C^QJ) be a wavelet transform coefficient at spatial location (/',/) in the LL 
subband, C\ (m,w) be a wavelet coefficient at spatial location (m t n) in the subband 
at level / (1**J,2,...) in orientation r (r=J,2,3J. Assuming that a 5-level wavelet - 
decomposition is used, the three type-3 ICUs defined by C u (iJ) can be represented 
20 in the following fashion. 

In orientation 1 : 

iC\&j)) U {Ci(2/,2 7 X C:(2/ + Uy), Cl(2i,2j + l),CU2j + \,2j + l)) (J ... 
In orientation 2: 

(Q 2 (/J)J U {Cjp/^ACjf^ + UA C 4 2 (2/,2^ + ]),C 4 2 (2z + l,2y + l)} u ... 
25 In orientation 3: 

{Cjfc./)} U {C 4 3 (2/',2y),C 4 3 (2/ + l > 27), C 4 3 (2/, 2y + l), C 4 3 (2/ + l,2 7 + l)} U ... 

Notice that the primitive C u (i f j) is not included in any the three ICUs. Instead, the 
LL subband is coded differently, using the type-2 ICUs. 
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Coding Type-1 ICUs 

Type-1 ICUs are defined exclusively for coding 8x8 DCT coefficient blocks. Coding 
of type-1 ICUs follows the JPEG baseline algorithms. 

Coding Type-2 ICUs 

5 Once a region is decomposed into a set of non-overlapping type-2 ICUs, each ICU is 
coded independently, and the collection of codes for all ICUs is the coded 
representation of that region. In this subsection we describe three techniques that may 
be used for coding type-2 ICUs. 

Using Embedded Quad tree 

10 Embedded Quad tree Wavelet (EQW), Figure 24 is an efficient and fast method for 
type-2 ICU coding. This technique implements an embedded progressive sorting 
scheme in a quad tree-like structure. In contrast to inter-band coding methods, the 
EQW explores the intra-band self-similarity of the wavelet decomposition 
coefficients. The EQW-produced code-stream realizes scalability by pixel-precision 

15 (the scalability by spatial resolution is realized by the multiplexer.) 

Depth-First and Breadth-First Quad trees 

The EQW method can be used for both lossless and lossy coding. In lossless coding, 
the primitives to be encoded in the ICUs are coefficients of a reversible wavelet 
transform. In lossy coding, after the wavelet transform, each transform coefficient is 
20 represented in a fixed-point binary format - typically with less than 16 bits - and is 
treated as an integer. 

In each ICU, the maximum coefficient magnitude M is determined. Then a value N is 
found which satisfies the condition 2 W < M < 2 N * X . The EQW works in a bit-plane 
manner: the initial bit-plane is set to 2 N , followed by 2* M ,2 A '~ 2 ... and so on. A 
25 binary significance map is produced for every bit-plane by comparing coefficients in 
power of 2 increments. 

Figure IV.3 illustrates the EQW encoding process on a single bit-plane. Two working 
lists are used. One is called the list of significant primitives (LSP). Each entry in 
LSP corresponds to an individual primitive in the ICU. The LSP is initialized as an 
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empty list. Another list is called the list of insignificant blocks (LIB). Each entry in 
the LIB is registered with the coordinates of the top-left corner of the block, together 
with the width and height information. Each LIB entry represents an individual 
primitive of width and height equal to 1. At the highest bit-plane 2 V , the ICU to be 
encoded is put into the LIB. A temporary list of insignificant blocks (TLIB) is used 
for refreshing the LIB after each bit-plane coding pass is completed. 

There are two methods used to add the four sub-blocks into the LIB that occur as a 
result of the quad tree decomposition. The first method, known as depth-first quad 
tree coding, is to insert the four sub-blocks into LIB at the position of their parent 
block. In this case, the four child blocks are evaluated immediately after the parent 
block. This rule is applied recursively until no more subdivision is possible. This 
means that control returns to the next entry in the LIB only after the present entry is 
completely encoded up to the highest resolution level. The second method, known as 
breadth-first quad tree coding, is to add the four sub-blocks under consideration to the 
end of LIB. In using the breadth-first process, all parent blocks at the same level will 
be processed first, followed by their respective children blocks. 

After all entries in the LIB have been processed, the entries in TLIB can be reordered 
according to certain pre-defined set of rules (this is an optional step.) Experimental 
evidence suggests that a higher PSNR can be achieved by using an effective 
reordering scheme. Then the LIB is replaced with the TLIB and the coding resumes 
at the next bit-plane. 

The next step is the refinement pass. The N 01 bit is output for entries in the LSP. 
Then, the process resumes using the new LIB and a new bit-plane set to Error! 
Objects cannot be created from editing field codes.. 

EOW with VLC 

In order to use variable length coding (VLC), the standard EQW algorithm of the 
previous section is modified as follows. 

Step 1 . Initialization: output n satisfying the inequality 2" < max{|C tf |} < 2"* 1 , set the 

list of significant primitives (LSP) as an empty list. Put the ICU to be coded into the 
LIB. 
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Step 2. Bit-Plane Sorting: for each entry of the LIB, perform quad tree coding. 

If all primitives within the block are insignificant, output a 0 bit and add this block to 

the temporary LIB (TLIB). 

Otherwise, look at the significance of the four sub-blocks. Then according to the 
5 VLC table, output the corresponding VLC sequence (see the table in Figure 25). If 
significant, sub-blocks that are not a single primitive are inserted into the LIB at their 
parent position. If insignificant, they are added to the TLIB. If the sub-blocks are 
single primitives, they are added to the LSP. Sign bits are output only if units are 
significant. If insignificant, the units are added to the TLIB. 

10 Step 3. Reordering (optional). 

Step 4. Refinement: for each entry in the LSP, except those included in the last 
sorting pass, (i.e. with same ti), output the mh most significant bit of jcj. 

Step 5. Quantization Update: decrement n by 1 and go to step 2. 

Some other researchers have also proposed the embedded coding methods where quad 
15 tree is used to explore the intra-band similarity I1]I2] . However, some major 

differences exist in how the quad tree operates. The work of [2] has suggested that 
empirically the quad tree decomposition should stop at size 16x16. However (as a 
result of our experience and experimentation) this size may not always be the optimal 
choice. Even without VLC, the efficiency of DAC's EQW is comparable with EZW. 
20 In fact, for a 2x2 block, if it is insignificant, one zero can represent four zeros. 
However, when a 2x2 block is significant, quad tree coding is inefficient. 

Using Block-Based Quantization 

The embedded quad tree (EQW, EQPC, etc.) is a special case of the more general 
block-based quantization techniques. Other block-based quantization algorithms, 
25 such as EBCOT, can also be very efficient for type-2 ICU coding. 

Using JBIG Algorithms 



F. Kossentini et al. "Embedded Quadtree Predictive Codec", ISO/BEC/JTC1/SC29/WG1/N667. 
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It is noted that in all implicit quantization schemes using progressive bit-plane coding 
methods, including the embedded zero tree and the embedded quad tree approaches, 
at each bit-plane, the ICU coder is actually dealing with a bi-level image (the 
significance map.) Therefore, by adopting those efficient JBIG algorithms as ICU 
5 coders in a JPEG 2000 system, one can realize two modes of interplay with JBIG. In 
the following EQW is used as an example to illustrate this idea. 

In the first mode, the transcode mode, the JBIG ICU coder and EQW ICU coder 
provide a two-way transcode between JPEG 2000 and JBIG (Figure 26). In this 
10 transcode, the EQW procedure is called for bi-level type-2 ICUs, with the total 
number of bit-planes being set to one. Note that the sign bit coding in the EQW 
algorithm should be skipped. 

In the second mode, the embedding mode (Figure 27), the proper JBIG routines are 
called as the bit-plane coder in the EQW routine for certain bit-planes and for certain 
15 decomposition sizes (the quad tree decomposition stops at this particular size and the 
coding is handed over to the JBIG routine). Since the algorithms for bi-level data 
coding are also evolving with the compression technology, having an embedding 
JBIG mode reserved in the JPEG 2000 system can make the Standard evolve with its 
sister technologies. 

20 Coding Type-3 ICUs 

Type-3 ICUs are defined to support various inter-band coding methods for wavelet 
transform coefficients. In particular, the methods of EZW, SPIHT, etc., are known to 
be efficient approaches for coding the type-3 ICUs. In testing these approaches, it is 
noted that the standard versions must be modified slightly to make them fit into the 

25 RICS architecture. Normally, these algorithms implement the ICU coding and 
multiplexing steps into an integrated procedure. A simple modification is required to 
bridge the existing architecture to the RICS architecture. The two steps must be 
separated, which is straightforward. Since the multiplexing stage in RICS can 
effectively simulate the performance of zerotree-based schemes (refer to Ch.V), this 

30 particular split in functionality has virtually no impact on the attainable coding 

(2J Taubman, D. et a). "EBCOT: Embedded Block coding with optimized truncation", ISO/EC/ 
JTCJ/SC29/ WG1/ NJ020R 
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efficiency for standard schemes. Instead it adds more flexibility and openness to the 
existing architecture. 

Intra-Region Coding with Different Types of ICUs 

Generally there is a need for using different types of ICUs for a intra-region coding. 
5 For example, in coding the wavelet transform coefficients, the LL subband is usually 
encoded differently than the other high-pass subbands. Because different sets of 
primitives may possess different statistical characteristics, providing several coding 
scheme choices in the intra-region coding module may offer a higher coding 
efficiency. In practice, the following combinations are useful. 
10 • In a wavelet decomposition, coding the LL subband with type-2 ICUs whereas the 
other subbands with either type-3 or type-2 ICUs. 

• Tiling an image into several regions, with some of the regions paved with type-1 
ICUs and others with type-2 ICUs. 

• Using different coding schemes in different type-2 ICUs. 

15 Intra-Region Coding without Anv ICUs 

Intra-region coding can be performed without specifying, any ICUs in a particular 
region. In this case, the entire region is considered an ICU. If the natural geometric 
shape of the region fits into any of the three ICU categories, the appropriate ICU 
coder can be used. If the region has a very irregular shape, then the ID coding 
20 method can be an efficient approach. Once the data for a particular region is scanned 
in a certain pre-defined order to form a ID stream, the following ID progressive 
sorting algorithms can be used to produce an embedded code-stream. 

ID Progressive Sorting Algorithms 

The EQW algorithm can be readily extended to ID cases where a binary partition is 
25 used instead of a quad tree decomposition: 

Let L- {c,} be the ID list to be encoded, LSP the list of significant primitives, LIS 
the list of insignificant subsets, TLIS the temporary list of insignificant subsets. 

Step 1. Initialization: Output n satisfying the inequality 2" < max{|c, |} < 2" 41 , set the 
LSP as an empty list. Put the set L into LIS. 
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Step 2. Bit-Plane Sorting: For each entry in the LIS, perform the binary partition. 
If all primitives within the subset are insignificant, output a 0 bit and add this subset 
to the TLIS. 

Otherwise, output the two bits that reflect the significance map. For those subsets, if 
5 they are not single primitives, insert them into the LIS at their parent position if they 
are significant, or add them to the TLIS if they are not. If the subsets are single 
primitives, add them to the LSP and output the sign bit if they are significant, or add 
them to the TLIS if they are insignificant. 

Step 3. Reordering (optional). 
10 Step 4. Refinement: for each entry in the LSP, except those included in the last 
sorting pass, (i.e. with sameTi), output the mh most significant bit of \c.\ ; 

Step 5. Quantization Update: decrement n by 1 and go to step 2. 

Processing with a Multiplexer Architecture 

15 The JPEG-2000 committee has been investigating many emerging compression 
technologies in order to define a new still image compression standard. The emerging 
standard will address both present and near future image compression needs. The task 
of selecting what technologies to include in the new standard is not easy given the rate 
at which technological advances occur in this area. The primary focus of DAC has 

20 been to develop a still image compression engine that addresses the issues set forth by 
the standards committee. 

The concept of using a multiplexer (MUX) has been adopted in some standardized 
processes of information coding such as MPEG. In contrast, incorporating a MUX as 
an integral part of a still image compression engine is rather unique, and there has 
25 been very little research conducted in this area. However it will be shown that the 
MUX concept is extremely useful in the development of a general purpose still image 
compression engine. 
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One of the concerns in developing a new Standard is the layout design of the encoded 
bit stream. A protocol must be developed to guide the new standardized compression 
procedures. Some of the important considerations in this area are listed below. 

5 • A well defined bit stream. All syntactic and semantic aspects of the encoded 
image format must be defined. The JPEG-2000 standard will encompass a wide 
variety of compression needs. One to one relationships will exist between fields 
that exist in the bit stream and functionality that exists in the core compression / 
decompression engine. The number of fields that exist in the bit stream is directly 
10 related to the overall functionality encompassed by the new standard. 

• A highly degree of data accessibility. The bit stream must be organized in such a 
manner that it need not be decoded in its entirety to interpret the physical meaning 
of the compressed data. Highly accessible data requires that the encoded data is 
divided into logical units each corresponding to a specific part of the original 
15 uncompressed information. The way this was accomplished in the existing JPEG 

standard was to partition the original image into 8x8 blocks. Each block was 
compressed using the DCT. Quantization tables were developed for coding the 
coefficients such that the reconstructed image quality was degraded in a standard 
fashion. However, strict block based data accessibility may not be a suitable 
20 medium for many of the current image processing needs. 

• Dynamic re-order ability. Dynamic re-orderability is a data access issue relating 
to the degree that encoded information can be reorganized to suite user specific 
needs for an image under consideration. This type of data access is an important 
concern for many applications, especially in the medical field. 

25 • Built in error resilient In any transmission medium there is always the 
possibility of incurring bit errors. The encoded bit stream must have a certain 
degree of error resilience to address this concern. 
Using a MUX to organize the encoded bit stream can address these issues. The first 
step is to divide the encoded information into logical groups. Organizing the data in 
30 ICUs leads naturally into a high degree of accessibility. Each ICU is an independent 
unit. The size and position information is self-contained within each ICU. One of the 
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effects of using a muliplexed bit stream design is that it is inherently erTor resilient. If 
any logical unit incurs an error, only that unit needs to be recovered. If the encoded 
data is divided into an array of logical units in preparation for MUX encoding, 
dynamic re-ordering is much easier to achieve. 

5 MUX Design Overview 

There are five important considerations to address in the design of the MUX for still 
image compression systems. 

• Defining a working model. 

• Defining an optimal data processing order. 

10 • Capturing the data to be compressed in a suitable data structure. 

• Defining a data priority to be used for compression. 

• Packing the compressed data based on data priority and processing order. 

A Working Model for the MUX Discussion 

In preparation for the MUX discussion, consider that a multi-resolution 
15 decomposition hierarchy of data has been obtained for an image to be compressed. 
Also consider that there are three channels of data. Assume that the original image 
has been transformed from RGB to YUV for example. In addition to this, assume that 
the U and V channels have been down sampled by a factor of 4:1. This type of 
approach is common for lossy image compression. The data sets appear in Figure 18. 
20 The diagram illustrates a 4 level multi-resolution decomposition for the Y-channel 
and 3-levels for each of the U/V-channels. The wavelet transform MALLAT 
decomposition is used here for convenience. 

This is by no means the only model that exists for the MUX design. DAC makes the 
distinction between region and non-region processing modes of operation in their 
25 design implementation. A hierarchy of list structures is used to control optimal bit 
budget distribution and flow of both non-region and region bit levels through MUX 
channels. The discussion presented in. the next Section begins by introducing the 
concepts in normal processing modes. The normal MUX modes of operation are 
generalized later on to include specialized region and mixed processing modes. 
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Non-Region Data Processing Orders for the MUX 

Generally speaking, given a wavelet multi-resolution decomposition of data, 
coefficients of a fixed magnitude contained in a lower resolution level are more 
important to the image reconstruction procedure than coefficients of a similar 
5 magnitude at a higher resolution level. This leads to a natural ordering of the 
coefficients beginning at the lowest resolution level. The natural ordering concept is 
an important consideration in the design of the MUX. 

General Color Processing Order (Lossy Case) 

Given the multi-resolution data sets of Figure V.l, a natural processing order exists 
10 for each decomposition channel. This natural order is illustrated in Figure 29, This 
particular order reflects a level priority inherent to each data set. 

Another two orders are obtained by simply interchanging orientation data sets in a 
given level. The difference lies only in the convention used for ordering the detail 
information (i.e. LH, HL, HH). The 4-1-1 down sampling relationship that exists in 
15 the color transform domain also exists in the wavelet decomposition data since there 
is one less transform level for the U/V channels. This is one of the keen design 
considerations that can be exploited in many stages of the MUX implementation and 
encoding / decoding procedures. The data is organized into list structures to cover the 
natural processing orders that exist in the data to be compressed. 

20 Level Priority Processing Orders (Lossy Case) 

The individual orders of Figure V.2 can be combined into a single natural processing 
order for the example under consideration. The new order is obtained by interleaving 
the expressions of Figure 29 using the inherent decomposition level priority of the 
data. The result is illustrated in Figure 30. 

25 Notice. that in this particular order, the 4-1-1 color down sampling relationship is 
maintained. In terms of the reconstruction process, the complete level 4 Y-channel 
information (i.e. LL4-HL4-LH4-HH4) is required together with the U and V channel 
low pass information (i.e. LL 3 in each case). This is the information the decoder will 
need first to reconstruct the low pass inverse color transform image at the lowest 

0 inverse wavelet resolution level. A "level split" on the U / V data channels at the 
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lowest inverse wavelet resolution level is used to exploit the natural relationships that 
exist for both inverse transform spaces. A similar order exists for the lossless case 
where full data sets are used. 

Color Interleave Processing Order (Lossy ) 

5 Another processing order that exists for color images in the wavelet transform domain 
is obtained by interleaving the color channel detail information. The order is 
illustrated in Figure 31. This type of ordering may be suitable for certain applications 
such as those that require the data to be encoded for a progressive download. As in 
the previous case, the down sampling relationships that exist are maintained in both 
10 color / wavelet transform spaces. 

Lossless Color Processing Orders 

As in the lossy case, similar processing orders can be designed for lossless color 
image compression where full data sets exist for all color channels. In this case, 
corresponding orientation information from each wavelet channel is related in the 
15 original inverse color space. These data should be processed and grouped together in 
the wavelet transform domain and eventually the final bit stream. 

The natural processing order for lossless color image processing is illustrated in 
Figure 32. The color interleave order for lossless image processing is illustrated in 
Figure 33. Note that in each case, the full decomposition data set for each channel is 
20 maintained. The inherent relationships that exist in the color transform space are 
maintained in the wavelet transform space for ordering and reconstruction of the 
original image. 

Capturing the Data in a MUX List Structure 

There are a number of candidate algorithms under consideration by the JPEG-2000 
25 committee for optimal quantization of the wavelet coefficients. A common approach 
used to quantize the wavelet coefficients in a progressive manner is to use an optimal 
bit plane ordering technique. DAC has testing a number of bit plane ordering 
techniques in both one and two dimensional schemes. Bit plane ordering techniques 
are generally classified as using implicit quantization for the wavelet data sets. 
30 Coefficients are packed into the final bit stream based on bit level priority. The bit 
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plane processing technology will be used to introduce the data structure used inShe 
MUX design. However, the MUX processing discussion that follows is not restricted 
to bit level processing algorithms. 

The general ordering and processing relationships are useful for introducing the MUX 
5 control architecture at a basic level. However, the key to the operation and many 
benefits of the MUX technique is in the data structure design used to encompass the 
natural ordering concepts. The first step is to define a list structure for each level, 
channel, and orientation of data that exists in the wavelet transform space. A typical 
example of this type of structure is given in Figure 34. 

10 A general list structure for each level data set can be utilized in many ways. A 
lossless "prepack" of multi-resolution hierarchy information is useful for exploiting 
the many relationships that exist in the data taken in MUX list processing context. 
The data and information contained in each list is used to control how the final bit 
stream is organized. All data packed into the final bit stream must conform to this 
15 structure. New fields may be added, but the basic operation of the structure remains 
the same. Most implicit quantization techniques are implicitly decodable. In other 
words, minimal header information is required for each list. The MUX architecture 
> organizes the lists or units of information into a bit stream that is scaleable in terms of 
bit precision and resolution and is controllable by the many MUX modes of operation. 
20 As an example, suppose that EQW (DAC's current two dimensional bit level 
processing design) is used to capture the leading ones and refinement bits at each bit 
level in each data set. Following the order given in Figure V.3, a multi-dimensional 
hierarchy of list structures is defined for use in the ensuing data processing stages. 
Within each list (as EQW progresses), the fields in the corresponding list structure are 
25 updated. The bit stream at each bit level is appended to the MUX "prepack" buffer as 
well as its corresponding size being added to the total in the bit packing information 
field. The bit level position where processing begins is put in the high bit information 
field. The total packed list size is placed in the total packed field. Finally if more 
than one internal processing scheme is used (e.g. NULL transform mode for bi-Ievel 
30 information or other internal modes of operation), the scheme used for the list is 
placed in the scheme field. 
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Each MUX list is fully independent, implicitly contains a full set of statistical 
information available for determining the amount of data to pack for each list and is 
optimal for organizing the final compressed bit stream. 

5 Data Priority and MUX Control 

The information contained in each list structure is used to organize the final bit stream 
based on end user compression requirements. DAC has implemented numerous MUX 
modes o operation. Two common modes of operation are outlined in this Section. 

Signal to Noise Ratio (SNIP Progressive Mode 

10 This mode of MUX places priority on the data such that the final ordering optimizes 
the SNR given a user specified image compression size. For the discussion that 
follows, assume that the user has specified the desired compression ratio in some 
manner. Assume that an optimal bit budget distribution scheme is in place. Thus the 
number of bytes that has been allocated to each color channel is known (more will be 

15 said on how to determine the optimal bit budget distribution for each channel using 
the MUX later). The information fields in the MUX list structures are used to 
determine the amount of data to pack for each list such that the SNR is optimized for 
the specified bit budget. A psuedocode explanation of this technique appears in 
Figure 35. 

20 Fields that refer to the MUX list structure appear in italics while other local variables 
appear in normal typeface. Basically the idea is to loop through each list checking to 
see whether the current processing bit level is equal to the bit level of the list under 
consideration. If the two bit levels are equal, the channel bit budget is decreased by 
the amount of data available in this particular list for the bit level in question (from 

25 MUX field pliPackinglnfofcCurBitLevelJ. The MUX field UCurBytesCount is 
incremented by the number of bytes available. Additional processing takes care of the 
remaining bits. The remaining bits will be considered in the channel bit budget on the 
next visit to this particular list. The MUX field cCurBitLevel is decreased by one 
such that it will again be enabled when the bit level in the main loop is decreased by 

30 one. 
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This procedure optimizes the SNR since it processes the largest bit levels in each list 
first. Thus information for the largest coefficients throughout individual 
decomposition data sets is sent first according the natural processing order outlined 
earlier. Once the channel loop exits, each MUX list will contain a field indicating the 
5 amount of data to pack into the final bit stream in each case. 

Resolution Progressive Mode 

The resolution progressive mode of bit budget distribution is a simple extension of the 
psuedocode implementation of the SNR mode of the previous Section, In order to 
obtain a resolution progressive mode, the 'Waveket Transform level' for loop is taken 
10 outside of the main BitBudget /BitPlane while loop such that resolution level is given 
priority. In this manner, a lower resolution image can be reconstructed in the inverse 
wavelet transform stage. The resolution of the ensuing image depends upon the 
number of levels required in the end user requirements. 

Data Packing Using the MUX Lists 

15 Composing the final bit stream is a simple matter once the bit budget distribution 
scheme has run to completion. The packing technique can follow one of the normal 
processing orders outlined earlier (e.g. color level priority or level priority color 
interleave) or another access technique as required by the end user. The amount of 
data to pack for each list has been determined in the bit budget distribution stage. If 

20 the data to be packed for a given list is greater than zero, then the total data size, 
scheme and high processing bit level are packed as header information into the final 
bit stream. If the data to be packed for a given list is zero (i.e. not required), then a 
smaller header is packed to indicate a zero length list. Currently a register type 
approach is being developed to handle zero length lists. All lists that have data 
25 available will set a bit position in the list register so that the decoder can determine 
which lists have made a contribution. Empty list will be flagged with a 0 bit in the list 
register. The list register is packed in the final code-stream. This technique will save 
the size of the list header overhead of empty lists. 
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Overhead Consideration of the MUX Architecture 

There is a small overhead placed the compressed data by the MUX implementation. 
There are three header fields to pack for each MUX list. 

• The total bytes packed. 

5 • The scheme used to process the list. 

• The highest bit plane for data in list. 

Note that if there is only one internal scheme used to process the entire image, then 
the scheme field does not need to be packed for each list. 

In order to calculate the cost of using a MUX in terms of header sizes for each list, 
10 consider that a 24 bit color image of size 2048 by 2048 needs to be processed in a 
lossy fashion (i.e. the working MUX model of Figure 28). Assume that the size of the 
smallest wavelet transform level is 8 by 8. Thus there are 8 decomposition levels to 
consider in forming the MUX list hierarchy. Assume that the wavelet decomposition 
data has been converted to a 16 bit integer representation. Also assume that there is 
15 only one internal processing scheme for the lists. Given these initial conditions, a 
worst case analysis is conducted to determine the header size for each list. The results 
are tabulated in Table V.I for the Y-Channel wavelet decomposition. 

From table in Figure 36, the total Y-channel header size for this particular image is 
443 bits. A similar calculation can be conducted on the U/V-Channels (assuming one 

20 less decomposition level for each) to yield 368 bits each. The original raw image size 
is 2048 x 2048 x 3 bytes. Thus the MUX packing overhead in this particular example 
is about 1 header bit for every 854 bits of compressed data. In the lossless case where 
each channels has the same number of decomposition levels (or correspondingly, in 
the 8 bit grayscale case), the overhead is still very small at 1 header bit for every 757 

25 bits of compressed data packed. 

The table in Figure 37 outlines the overhead size for square images with dimensions 
that are a power of 2 beginning at 16 by 16 and ending at 64k by 64k. Note that the 
overhead size if 44 bits in both the lossy and gray scale cases. The reason for this is 
that the wavelet transform is not used on the U/V channels when the original image 
30 dimensions are 16 by 16. The Y-channels is decomposed once. A plot of percentage 
overhead versus image dimension is given in Figure 38. Both lossy and lossless cases 
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appear on the same plot. Note that, the image size can be quite small while still 
maintaining a relatively low overhead in terms of the total MUX list header sizes. 

Bit Budget control from the MUX Architecture 

One of the most difficult tasks that must be addressed in any compression scheme is 
5 scalability. The internal procedures must be developed such that the image under 
consideration can be compressed to a user specified size in an optima! manner. The 
task is compounded for color images since there are three channels to consider. The 
major problem is how to distribute the user specified bit budget between the three 
channels. 

JO Standard color transforms such as YUV or YIQ are normally used to redistribute the 
RGB color information prior to the multi-resolution decomposition stage of the 
compression procedure. These transformations concentrate most of the energy into 
one channel making them better suited for compression. Another color transform that 
is gaining popularity is the KL transform. This particular transform technique is 

15 based upon a principle component analysis (PC A) implemented on the original raw 
color information in order to determine the optimal color redistribution for the three 
RGB channels. Generally speaking, after completing a color transform stage, two of 
the three channels are down sampled by a factor of 4 to 1 for lossy compression. In 
doing this, the amount of data that must be compressed is reduced by a factor of 2 

20 with minimal loss in visual quality for the reconstructed image. 

The easiest bit budget distribution scheme to implement is one that follows the color 
transform down sampling ratios. If the transform under consideration is YUV-411, 
then 4 parts are allocated to the Y-channel for every 1 part allocated to the U / V- 
channels respectively. However, the energy distribution of the wavelet transform data 
25 generally does not follow this strict down sampling guideline. Instead it varies from 
one image to the next. 

DAC has developed a dynamic bit budget control architecture that is based on the 
implicit information contained in each MUX list. In this Section, a bit budget 
allocation procedure will be introduced that can be used for optimal scalability in 
30 normal processing modes of operation. This technique is based implicitly on the 
amount of energy contained in each color channel of the wavelet decomposition data 
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sets. A data ratio concept is used together with the prepack information contained in 
the MUX list structures to determine the budget for each channel. 

Compression procedures that process wavelet coefficients in a bit plane order are 
implicitly tracking the energy contained in the decomposition. The most significant 
5 bits (MSBs) of the largest coefficients are packed first followed by refinement bits 
and the MSBs that are significant for coefficients at the next bit level. This process is 
repeated in a recursive fashion until the desired compression size is obtained. 

Let yij , Uij and Vy , be the total data available in a particular orientation level for the 
three color channels. Then the total amount of data available in the Y-channel 
10 decomposition Y t is obtained by summing the individual totals. A similar procedure 
can be followed to obtain U ( and V t . 



15 



N 3 

Y t = yt* ■ + 2 2 y« 

N 3 

U, = ukm + J £ ug (Eq.V.l) 

N 3 

V t = vrw + 2 £ Vy 

i=i ri 



25 

The next step is to calculate the pack ratios to be used for each channel of the wavelet 
decomposition hierarchy. The pack ratios are determined by taking the ratio of the 
two largest amounts of data to the smallest amount of data. Let the three pack ratios 
be denoted R y , Ru and R v . The smallest data size will be in either the U or V 
30 channels because of the down sampling step. Note that one of the pack ratios will be 
unity. 

Y, 

35 R, 



MJN(U,,V0 
Rv ~ MIN(U b V0 
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The pack ratios given in Eq.V.2 are used to determine the optimal amount of data to 
allocate for each color channel based on the user specified compressed file size. 
However, any additional overhead introduced into the final bit stream by the MUX 
5 must be taken into consideration. Let yhy, uh y and vhjj be the individual header sizes 
in each wavelet channel for a particular orientation level. Then the total header sizes 
for each wavelet channel are obtained by summing the individual totals. 

N 3 

Y h = yh N 4 + £ £ yhij 
N 3 

U h = uh w + £ 2 uhs (Eq.V.3) 
t-i j-i 

N 3 

Vk = vh H < + £ £ vhy . 
1=1 J=l 



15 



20 



If doing ROI processing, the packing overhead introduced by the mask must be taken 
into consideration in determining the bit budget for each channel. Let the total pack 
25 size for the ROI mask be denoted as M t . The logical approach to take is to make an 
adjustment in the bit budget for each channel based upon the pack ratios of Eq.V.2. 
In this manner, the mask overhead is distributed proportionally to each wavelet 
channel. The unit adjustment factor U. is determined from the total mask overhead 
and the total pack ratio. 



Mt 

U = (Eq.V.4) 

35 Ry + Ru + Rv 



The adjustment size for each channel is determined by using the result of Eq.V.4 and 
the pack ratios of Eq.V.2. 

Yi - Ry*U. 

= Ru-u. (Eq.V.5) 
Vi - Rv-U. 54 
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10 



A useful set of calculations that comes out of Eq.V.l to Eq.V.5 is the minimum 
compression bits per pixel (BppMin) and the maximum compression bits per pixel 
(BppMax). These values are calculated below (assume that the original raw image file 
size is F t bits). 

Bppm« =— • (Y. + U1 + Vt + Yb + Uh + Vh + Y. + U.+ V.) 
Ft 

« . (Eq.V.6) 

BppMin =- a ~ ' (Yh + Uh + Vh + Y. + U. + V.) \ m / 



These expression represent bounding compression values available for the image 
20 under consideration. If all data is packed for each of the wavelet data sets, the result 
is BppMax If just the header information is packed, then the result is BppMin- 

Suppose the user specifies a compressed file size of F UMr . The unit bit budget Uw> is 
determined from the pack ratios and the total header sizes. 

25 

fi-(Yh + Uh + Vh) ^ 
Ubb = 1 — (Eq.V.7) 

Ry+Ru + Rv 

30 

Now the optimal bit budget for each channel Ybt>, Ubb and V b b can be determined 
using Ubb, the individual pack ratios and the overhead parameters. 

Ybb ^ Ubb. Ry + Yh-Y. 

Ubb = U»*- Ru +Uh-Ui (Eq.V.8) 

Vbb = Vbb. Rv + Vh- V. 

40 

The expressions given in Eq.V.8 will always yield a near optimal bit budget for each 

color decomposition channel. The technique is based implicitly on the energy 

distribution map recorded in MUX lists and the total overhead associated with the 
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underlying process. Note that if processing with no ROI, then that variable falls out 
of the calculation. 

This concept can be extended to form pack ratios for each subband if more accuracy 
is required. A similar data ratio technique can be implemented to control the bit 
5 budget in that case. The necessary information is contained in the MUX list 
structures. However for most practical application, the cited distribution technique is 
sufficiently accurate. The user specified file size can be obtained within several bytes 
in implementing this technique. In addition, the maximum and minimum attainable 
file sizes are known prior to the final packing stage. 

10 A simple routine has been developed for distributing the final bit budget in the level 
where it is determined it will expire. The idea is quite simple. Given that the bit 
budget will expire on a particular bit plane in a certain transform level, then the bit 
budget is redistributed such that each orientation gets a proportional amount based on 
the amount of data that each orientation can take. The amount of data each 

15 orientation gets is determined by using ratios between orientations on the bit level 
under consideration. This is an important consideration since the net effect is 
approximately a 1 dB improvement in the PSNR. 

Using the MUX for Region Processing 

So far the focus has been on the design of a MUX control architecture for normal 
20 image processing modes of operation. The main highlights are the data processing 
orders, the MUX list structure, the bit budget control technique, and organizing the 
final bit stream. DAC has extended the MUX concept to include a variety region 
processing modes of operation. 

Region Processing Orders 

25 The processing orders introduced in the previous Section are specific to the normal 
processing modes where one or more ROI are not specified. DAC has implemented a 
region processing design based on the natural extension of the MUX concepts 
introduced in the previous Sections. Both lossy and lossless processing orders are 
again considered. 
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General Color Region Level Processing Order 

The region level processing order places priority on each ROI in a descending fashion 
followed by the normal level priority scheme of the original MUX design. Inherent in 
the MUX scheme is the general assumption that data of similar magnitude in each 
5 succeeding region is less important to the overall image reconstruction. A secondary 
priority key is placed on the wavelet transform level. Data of similar magnitude 
contained in a lower resolution level is more important to the reconstruction 
procedure than data at a higher resolution level. The general processing order for 
each color channel (assuming a 4 region priority scheme) is given in Figure 39 where 
10 R* is the region level for i = 1,2,3,4. 

Region Level Color Processing Order (Lossy') 

The individual orders of Figure V. 10 can be combined into a single processing order. 
The new order is obtained by interleaving the expressions making use of the inverse 
color and wavelet transformation relationship that is presented for the normal MUX 
15 modes of operation above. The result is illustrated in Figure 40. 

The intrinsic 4-1-1 color down sampling relationship is maintained for region 
processing MUX modes. In terms of the reconstruction process, the complete level 4 
Y-channel information (i.e. LL4-HL4-LH4-HH4) is required together with the U and V 
channel low pass information (i.e. LL3 in each case) for each region under 
20 consideration. 

Color Interleave Region Level Processing Order (Lossy) 

The color interleave processing order extends naturally to the MUX region processing 
modes. The order is illustrated in Figure 41. This type of ordering may be more 
suitable for certain applications that require the data to be encoded for a progressive 
25 download based on regions of importance. As in the previous case, the down 
sampling relationships that exist for the inverse color and the wavelet space are 
maintained. 

Region Color Processing Orders (Lossless) 

As in the lossy color region processing MUX modes, similar processing orders can be 

30 outlined for the lossless case where full data sets exist for all color channels in each 
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region. In this case, corresponding decomposition level orientation information in 
each channel is related in the original color transform space for each region under 
consideration. These data are processed and grouped together in the wavelet 
transform space and are ultimately put in the final bit stream by the MUX control 
5 architecture. 

The processing order for lossless color region processing is illustrated in Figure 42. 
Note that in each case, the iiill decomposition data set for each channel is maintained. 
The inherent relationships that exist in the inverse color and wavelet space are 
maintained for each region channel. 

DAC has developed both SNR progressive and resolution progressive MUX modes of 
operation for region processing. In addition, several specialized MUX modes were 
developed that may be useful in certain applications. The regions can be defined 
automatically by the technique described in Chapter III, or user defined ROI can be 
used to group the data. User defined ROI can have arbitrary shape or can be defined 
in terms of simple geometric elliptical or rectangular region primitives. 

One of the specialized MUX region processing modes can be introduced at this point 
since it fits into the context of the ordering discussion. 

Transparent Region Color Processing Orders (Lossy'* 

This particular region color processing order is useful for applications that require 
20 transparent region channels that has little or no influence on the ordering of the data. 
Either of the normal SNR and resolution progressive modes are overlaid with a 
complete region channel description. The technique is implemented simply by taking 
the region index to the inner processing loop in the pseudo code implementation 
example of Figure V.8. This ordering technique may be useful in video processing 
25 applications where a complete region mask description is required together with a 
compressed frame of information. The region description may be used to process 
subsequent frames in the sequence. One of the processing orders that falls into this 
MUX mode of operation is illustrated in Figure 43. In this case, the high bit plane of 
each region list is processed first causing the region classification of all coefficients to 
30 be transparent to the distribution and packing routines. 
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Transparent Region Color Processing Orders (Lossless) 

Corresponding transparent region processing modes exist for the lossless case. A 
typical example is given in Figure 44. As in the lossy case, the final bit stream is 
organized independent of the constraints of the region channels which may be useful 
5 in certain cases. 

MI DC List Structure for Region Processing 

The MUX organizational list structure concepts introduced for normal processing 
modes have been extended to include many region processing modes of operation. In 
this case, the total number of lists is a function of the number of ROI. If there are 4 
10 ROI, then there are 4 times as many MUX lists. However, the basic operation of the 
MUX control architecture is similar in each case: 

There are three high level modes used to categorize the MUX architecture. 

• Normal processing mode (ROI disabled). 

• Region processing mode (ROI only). 

15 • Mixed processing mode (normal and ROI enabled). 

The next Section outlines the operation of the bit budget and MUX controls for many 
of the region processing modes developed at DAC. The mixed processing modes are 
briefly discussed later in the Chapter. 

Bit Budget Control for Region Processing MUX Modes 

20 Transparent Region Level Color Mode 

Transparent region processing is mentioned above. In that particular ordering 
example, the region index is placed in the inner most loop in the bit budget 
distribution function. The processing index placement is basically a method of 
incorporating a transparent region layer of processing into the normal MUX modes 
25 outlined earlier in the Chapter. Processing and packing in this mode gives ROI a low 
priority. 

Distributing in this context ensures that the SNR progressive and resolution 
progressive MUX modes operate as they did before. The exception in this case is that 
there is a variable length region mask overhead that must be taken into consideration. 
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The bit budget distribution functions are calculated as they were for the normal JvftTX 
processing modes. Based on the file size (or resolution level) requirements, each 
color channel receives a proportional amount of the bit budget based on the ratio 
distribution technique used in the MUX architecture. 

5 The DCT auto regions and arbitrary or primitives user defined region types can be 
operated in this mode. In DAC's internal processing architecture the user must select 
the region coverage technique used to categorize the wavelet coefficients. DAC has 
implemented the wavelet mask down sampling technique of the JPEG-2000 VM, and 
it can be used in any of the region processing MUX modes. In this manner a common 
10 mask can be used in each orientation level or the VM mask down sampling technique 
can be used for individual orientation level mask coverage, for both automatic or user 
defined ROI in lossy and lossless MUX modes of operation. The number of region 
categories can be selected as 2, 3, or 4 channels. 

In order to test the transparent mode of operation the YUV-411 color transform is 
15 applied to a 24 bit 256 x 256 Glacier park mountain scenery image. The 9-7 kernel 
implemented in a lifting scheme is used as the wavelet transform. The DCT region 
formation technique of Chapter m is employed to generate the common masks for 
each wavelet transform level. The mean square error (MSE) is measured for a 
number of compression ratios. The result is illustrated in Figure 45 in a plot of MSE 
20 versus Bpp. The plot shows the break points in MSE for each automatic ROI used in 
the example. The rate of image degradation (i.e. MSE and PSNR) must be controlled 
precisely for each ROI. 

The MUX overhead is calculated based on the mask type and process selections. For 
25 arbitrary user constructed masks, the mask file can be loaded to guide the MUX. In 
the 4 ROI common mask case, the overhead is 0.5 Bpp in packing the entire mask; In 
the VM mask case, the overhead is 2.0 Bpp. If simple primitives are used to form the 
user mask instead of an arbitrary shape, the overhead is greatly reduced. The arbitrary 
mask overhead can be reduced with the addition of an entropy coding stage. The 
30 common DCT mask overhead is cited in Chapter ID as approximately 1Kb (0.2 Bpp) 
for an 8 bit 256 x 256 image size. These are rough estimates that do not take the 
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header sizes into account. However, they do serve as a comparative guideline for the 
current discussion. 

Region Priority Level Color Mode 

In this mode of MUX operation, the pack ratios for each wavelet channel are 
5 determined as in the normal processing mode. However, there is one important 
difference. In this particular mode of operation pack ratios are determined for each 
region channel. This implies that there are 12 pack ratios for a 4 ROI process. The 
bit budget distribution is based on the overall wavelet channel pack ratios and the ROI 
data totals in each region channel. The distribution function also takes the mask and 
10 list header size information into account in the overall calculation. 

This mode of MUX operation is designed to distribute the MSE and the PSNR image 
reconstruction measurements in an approximately uniform manner for all region 
channels. A proportional amount of overall bit budget is allocated to each region 
channel based on the region and color channel pack ratios. Figure 46 illustrates the 
15 result of using this mode of operation for the Glacier park image. Notice how the 
quality of each region channel degrades in comparison to the others. Auto detected 
DCT common masks are used to generate the result. 

Absolute Region Priority Level Color Mode 

20 In this mode of operation, absolute priority is given to the region channels. The 
distribution function is the same in this case in that the bit budget is divided into 12 
according to total data ratio technique mentioned earlier. However the bit budget is 
distributed in a biased fashion giving the highest priority to the most important ROI. 
Some regions may not receive a budget based on the compression requirements. The 

25 bit budget is distributed region by region beginning at the most important ROI. Thus 
complete sections of the original image can be eliminated altogether if the bh budget 
is small. If only the important regions of an image need to be saved, this technique 
can be employed to partition the image. 

A plot of MSE versus Bpp for the absolute region priority color MUX mode is given 
30 in Figure 47 for the Glacier park image. Notice how the regions fade out very fast 
and sharply. This occurs when the region no longer has any budget allocated to it. At 
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that point, the region is basically invalidated in terms of any contribution to the 
reconstructed image. That portion of the image basically fades out. Note that the 
amount of quality of the fade out depends on the down sampling technique to translate 
the masks. The VM mask formation technique causes a very gradual fade out to 
occur. In the common mask approach, the effect is much more abrupt. 



Scaled Region Priority Level Color Mode 

In this mode of MUX operation degradation rate of each ROI can be controlled. Each 
ROI contributes to the final bit stream. However, instead of having each ROI degrade 
at a uniform rate (as Section V.5.3.2.), the quality measurements of each succeeding 
10 can be controlled. The initial bit budget for each region channel is calculated as 
before in that the specified compressed file size is split into 12 bit budgets to be 
spread between each of the 4 region channels. However after the initial split, the 
allocation amounts are changed heuristically based on a priority factor that is set for 
each ROI. For example, suppose it is decided that ROI 1 is 50% more important than 
15 ROI 2, ROI 2 is 30% more important than ROI 3 and ROI 3 is 20% more important 
than ROI 4 (the background). Using this assumption as a starting point, the amount of 
data allocated to each region channel is changed slightly. The net effect is to cause 
the quality measurements in each region channel to degrade at slightly different rates. 
A plot illustrating this effect is given in Figure 48. Note how the MSE break points 
20 for regions 2 and 3 have shifted slightly towards the left. 

The result of using this particular MUX mode illustrates an important property that 
should be available for any ROI processing technique. The problem with many ROI 
processing techniques is that it is difficult to control how much each region 
contributes to the final bit stream. The technique outlined here can be used not only 
25 to implement this effect, but also to control the degradation for each region channel 
based solely on an importance factor associated with each ROI. 

Region Percentag e Priority Level Color MnHe 

In this mode of operation the user can specify a data percentage for each region based 
on the total amount available in each region channel. The distribution function is 
30 slightly different in this case. The data totals are determined for each region channel 
along with the wavelet color channel totals. There are still 3 bit budget for each 
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region channel. In this case however, the amount of data to include for each region 
channel can be set by the user as a percentage of the total in each case. 

Currently when this mode, of operation is invoked in DAC's compression engine, the 
user can cycle through the operation until the desired size for each region is obtained. 
5 The data totals are displayed after each run. Note also that in using this mode of 
operation, one or more ROI can be eliminated or faded heavily as in the absolute 
region priority MUX mode of Section V.5.3.3. 

A plot illustrating the result of using this mode of MUX operation is given in Figure 
49. Initially all available data is included for each region followed by equal 
10 decrements for each region channel. 

Using the MUX for Mixed Processing 

So far both normal (non-region) and ROI processing modes have been discussed. In 
addition to these modes of MUX operation, DAC has developed mixed mode 
processing capabilities. The modes of operation discussed for both normal and ROI 
15 processing are extended such that they can be run simultaneously for an image under 
consideration. A wavelet transform level partition is conducted based on the desired 
number of region and the number of non-region levels. 

Currently non-region levels can be defined for processing lower resolution levels and 
region levels can be defined for process higher resolution levels. However, the 

20 implementation is not restricted by this distinction. It can be changed to regions over 
non-regions or to an interlaced combination of the two. The parameter that controls 
this distinction is termed the region start level. It can be set to any valid wavelet 
transform level (it is set internally to -1 to disable region processing.) Thus the MUX 
technology can be used in exclusive non-region mode, exclusive region mode or a 

25 combination mixed mode of operation. 

The ordering techniques outlined for normal and region processing MUX modes will 
not be duplicated in this Section. The same concepts apply for mixed mode 
processing. The bit budget distribution functions work as they did before, but in this 
case they exist simultaneously. In some modes of operation there may be as many as 
30 15 bit budget definitions used to organize the final bit stream. The method used to 
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determine them has not changes. The same ratio technique that has its basis in the 
MUX list structure is used to determine the appropriate allocation in each case. 

There is one additional feature that can be exploited for mixed mode processing. An 
importance factor can be attached to the non-region levels. The net effect of this 
5 parameter is to taper the amount of data included for the non-region levels. In the 
current implementation, the non-region levels are processed first. Thus they 
considered first by the distribution function. The importance factor allows the user to 
decrease the amount of data included for the non-region levels by a certain 
percentage. The delta amount is considered in the bit budget distribution for the 
10 region levels. 

The region processing overhead is slightly smaller in this case. Depending on the 
region start level parameter, there will be less region header information required in 
the bit stream since there are fewer region levels. However, the header overhead for 
the lower resolution levels is quite small anyway. 

15 One example is given here to illustrate the operation of mixed mode processing (see 
Figure 50). In this particular case SNR progressive mode is used for the upper 3 
levels and scaled region priority on the bottom 3 levels. Thus the bottom level 
degradation rates have been adjusted to favor the most important region and 
attenuating in some fashion between the other 3 regions. Notice how the MSE is 

20 lower in this case with the same region overhead as the previous case of Figure 48. 

DCT Region Form ation as a Classification Scheme 

In observing this result, it is apparent that for the same region overhead, a better result 
is obtained in mixed mode. There are a number of reasons for this. The first is that 
the region channel coverage is not an exact overlay. All masks used to group or 

25 categorize data must deal with the down sampling issue at different resolution levels, 
and the tight overhead restraints of the compression channel. As it appears in the 
results presented here the DCT region detection mode can be applied to threshold 
wavelet coefficients to form region channels. The amount of data packed for each 
channel can be controlled by the MUX. In addition the initial data split used to create 

30 the partition can be controlled before the DCT mask procedure begins. 
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There is some region migration or intermingling of the coefficients that occurs at the 
boundaries where sharp changes occur in the original image. Some work was 
conducted in developing heuristic techniques to decrease the miscoverage in hot 
areas. And there is a benefit there. Furthermore the DCT low pass filtering stage 
5 affects the original priority bias used to partition the data. There will be a benefit in 
determining the optimal data split. By adjusting the original region partition (e.g. 
equally spaced regions), the plot of MSE versus Bpp can be set to partition the data in 
other ways. Inter-resolution coverage is another problem to consider. The mask 
generation technique of the JPEG-2000 VM addresses the miscoverage problem. 
10 DAC has completed some initial investigations of the VM down sampling technique 
and further testing is required. 

There is a trade off between accuracy of each region channel and mask overhead 
required for region channel operation and generation of the accuracy in the first place. 
The DCT approach shows much promise especially on the highest resolution level 
15 where the masks are the most accurate. One of the benefits of using this the DCT 
approach is that it can be translated to other transform levels in the frequency domain. 
Or alternatively, the new masks could be generated at a different wavelet level. 

Other Processing Modes 

There are many other modes of region processing operation that have not been 
20 presented in this document. The whole user defined region generation and MUX 
modes were not included. The processing modes introduced in this Chapter can be 
used in the same way to control each region channel. Simple primitives are cheap in 
terms of overhead. One mode of operation supports full mask sets so that arbitrary 
mask can be loaded and sent with the data to form the region channels. The mask 
25 coverage technique can be selected as common masks (a smaller raw size that is up 
sampled at the decoder side) or VM masks, which retain the original image 
dimension. More time is required to study the effects of both techniques as well as 
other region formation schemes. 

Experimental results for the resolution progressive modes of operation are not 
30 presented in this document. These modes of operation are not currently available. 
However their implementation is not difficult. These MUX modes are currently 
under development. 
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The experimental results presented in this Chapter were obtained using DACs ID bit 
level sorting implementation. DAC is currently incorporating the 2D version based 
on EQW into the region processing channel. There may be benefits in retaining both 
techniques. For example, in bi-level image processing. Currently the EQW sorting is 
5 implemented for normal (non-region) processing MUX modes. Both sorting modes 
are available for lossless compression. There are numerous wavelet kernels and color 
transforms for both lossy and lossless compression. In addition, the number of region 
channels can be set with a current maximum of 4. Primitives can be used as desired. 
The number of primitives is not restricted to 4 with region overlaps given to the most 
10 important region. 

Bit Stream Syntax 

DAC's current bit stream structure is rather dynamic given the different types of 
organizational strategies that exist in the underlying core technology. Normal, region, 
and mixed processing modes in addition to arbitrary wavelet, color, entropy coding 
various sorting stages in both lossy and losses cases. 

Currently the core architecture can be divided into 2 categories. 

• Lossless compression 

Full data sets lossless color selection, lossless integer lifting scheme wavelet 
types, lossless sorting and packing stages, with additional entropy coding can be 
selected for completely lossless and slightly lossy (some of the color transforms 
are slightly lossy) image compression. 

• Lossy compression 

In the lossy case, color selections, more wavelet kernels / lifting schemes, lossy 
sorting and packing stages, with variable length coding can be selected 

In both lossy and lossless cases region and mixed modes can be used. Currently 
normal operational modes can be realized in an almost transparent fashion for many 
current compression schemes as well as our own internal EQW / ID sorting. The 
general form for the lossy and lossless header structures is given in the tables in 
30 Figures 52 and 53. 
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The current architecture is very flexible for introducing new technologies. Many 6? 
the modules are completely interchangeable. The exact bit stream syntax depends 
largely on the mode of operation selected by the user (i.e. regions, no regions, sorting, 
lossy/lossless etc.) Generally normal processing modes can be selected for all 
5 transform levels, or for any number of lower resolution transform levels. Region 
levels can be defined for all transform levels, or the higher resolution levels used in 
combination with the lower resolution normal levels. At some future date, the region 
and the normal levels can be mixed. 

Each unit list organized by the MUX has a header tag. This header tag carries the list 
10 size and the high bit plane processing level for the list. The current implementation 
uses 5 bytes per list. However bit packing is currently under implementation for tag 
headers. This will reduce this by a significant amount. Only 2 packing schemes are 
used at the global level so that no additional tag header information is currently 
required. A diagram illustrating the structure of the tag header is given in Figure 53. 
15 As other core technologies are added to the core engine or the core technology 
advances, other fields may be required in the tag headers. 

The basic structure for normal modes of operation is given in Figure 54. The diagram 
illustrates a lossy pack arrangement for a color image (YUV down sampled color 
transform assumed) according to the packing/processing order given in Figure 30 
20 above. In this case the code stream consists of the file header, followed by the header 
tags/data. 

The basic structure for region processing modes of operation is given in Figure 55. 
The diagram illustrates a lossy pack arrangement for a color image (YUV down 
25 sampled color transform assumed) according to the packing/processing order given in 
Figure 40 above. In this case the code stream consists of the file header, region 
header, regions description and finally the header tags/data. 

The basic structure for mixed processing modes of operation is given in Figure 56. In 
this case the first item in the code stream is the file header followed by the normal 
30 processing mode header tags/data. Next in line is the region header, region 
description followed by the header tags/data. Notice that there is a region start level 
parameter 'r* in this case. Since there is one less transform level for U/V channels in 
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the lossy case, there is a processing shift in the U/V channels. This makes reference 
to the "level split" notion discussed in Chapter V whereby the natural 4:1:1 
relationship that exists for inverse color/wavelet transforms can be maintained for 
internal processing and the final code stream. 

5 

Transcodability 

In order to illustrate the generality and openness of RICS, two examples are shown to 
depict how the 'transcode' with bi-level images (JBIG) and JPEG can be handled in 
RICS. 

10 JPEG transcode 

Figure 57 shows how a DCT-based code can be produced in RICS. With this 
transcodability, it will be straightforward to write a small conversion program to 
transcode an old JPEG file into a JPEG 2000 file. Although it is possible to transcode 
a JPEG 2000 file into a JPEG file (losing some scalability), it will probably be of very 
15 little use. 

VJBIG transcode 

Figure 58 shows the execution path illustrating how a bi-level image such as a text 
document can be efficiently handled in the RICS system. The NULL transform is 
applied (nothing has to be done). Generally binary text documents contain mostly 
20 high frequency energy. Multi-resolution decompositions will not necessarily be a 
suitable basis for efficient coding. In fact, current well-known techniques specialized 
for binary images are applied directly in spatial domain. While using a RICS system 
for processing a compound document with mixed grayscale/color/text information, a 
set of rectangle shapes suffice to enclose the text regions. This is roughly equivalent 
to run-length coding in existing binary image compression techniques to skip strings 
of zeros. The pixels within each rectangular region are then coded into a one- 
dimensional stream via 1-D algorithms or JBIG routines, as discussed above. 

Post Processing 

30 For reconstructed images with wavelet based coding methods at low bit rate, de- 
ringing is usually a useful post processing procedure to improve (mainly) the visual 
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quality. Nonadaptive procedures, which apply the de-ringing filtering to al! pixels 
without discrimination, have the following problems. 

• While they can successfully remove ringing artifacts, they may also wash out 
details of image. 

• They usually rely on too many parameters that are to be determined by human 
users. 

• The process is time consuming. 

The RJCS system employs an adaptive de-ringing algorithm for post processing. 
Since the ringing artifacts usually appear around edges where sharp changes occur, 
this adaptive filtering process is applied only to edge areas. As the result, the post 
processing removes artifacts around edges and prevents fine details from being 
smoothed out by the filter. Compared with non-adaptive methods, the processing time 
is remarkably reduced since the filtering is applied selectively to pixels. The diagram 
of Figure 59 shows the algorithm of the adaptive post processing filter. First of all, the 
pixels of reconstructed image are classified into edge pixels and non-edge pixels. 
Edge pixels are enlarged into edge regions since the ringing artifacts do not occur 
right at the edges but around the edges. Then the artifact removal filter is applied 
only to the edge areas. 

We did some tests on this adaptive de-ringing procedure. For the filtering stage we 
20 used the filter described in the JPEG 2000 VM 3.0 (B). Figure 60 shows the result of 
the modified post processing. Artifacts are clearly visible in the first picture especially 
behind the cameraman's back and around the tripods. As we can see from the second 
picture, artifacts are removed, however, the details in the grass field and on the man's 
pants are also removed. In the last picture, the modified filter successfully removes 
25 artifacts and at the same time retains the details. Figure 61 shows the edge area (in 
white color) that was used in producing the third picture. 

As the result of using this adaptive filtering, processing time is greatly reduced since 
less pixel are processed. The table in Figure 62 shows the performance comparison of 
the VM 3.0 (B) post filtering and the RICS adaptive filtering 

30 Edge Detection 
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To determine if a pixel is an edge pixel, the average power difference between the 
pixel and its neighboring pixels are measured against a threshold (bottom threshold). 
If the average power difference of a pixel is above this threshold, the pixel is marked 
as an edge pixel. The threshold acts as a high pass filter that filters out the pixels with 
5 low power levels (Band pass filters can also be used for edge detection by introducing 
a proper ceiling threshold). 

Edges are thickened to form edge areas after the edge pixels are detected. With the 
right choice of the threshold, this modified filtering process can be applied to 
reconstructed images with compression ratios as low as 8:1. Since for most wavelet 
10 coded images, no artifacts can be visually detected for compression ratio lower than 
8:1, our results suggest that this filter can be applied to most reconstructed pictures 
regardless of its compression ratio. 

Parameters Optimization 

One of the problems with Shen's filter is that there are many parameters associated 
15 with the filter, which makes the filter difficult to use. Reducing the number of 
parameters will improve the usability of the post processing filter. 

There are three different potential functions implemented in VM 3.0 (B) for 
controlling the post processing: Quadratic Truncation, Huber, and Lorenzian. Our test 
results shown that the potential functions Huber and Lorenzian do not outperform the 
20 Quadratic Truncation in any of the 62 cases for PSNR measurements. Furthermore, 
visually Lorenzian creates many noticeable ghost images and shadows, and Huber 
does remove artifacts successfully however it blurs the image more than Quadratic 
Truncation. 

Computation Speed 

For evaluating the three potential functions, the following computation needs to be 
calculate nx(n-l)f 2 times for every pixel (where /? = filter size x2-l). 

Lorenzian: 1 float division, 2 float multiplication, 1 float addition and 1 

float 

log operation 
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Huber: 1 comparison, 1 float multiplication or 2 float multiplication 

with 1 

addition 

Quadratic Truncation: 1 comparison and 1 float multiplication 

5 In our tests, Quadratic Truncation is the fastest and it performs the best for artifact 
removal with the least degradation to image quality. Therefore, the other two 
potential functions will not be used. 

Threshold (y parameter in the potential function) 

The y parameter is examined using the quadratic truncation potential function (default 
10 value is 16), given by: 

e = argmin J^p(X t -Xj) 

We can see from the equations that only pixels with similar neighboring pixels are 
affected when varying y. When y is decreased, there will be more y 2 as the result of p 
15 and the intensity of similar pixels output image will be more uniformed resulting a 
more blocky looking type of image (less gradual) for regions with similar intensities. 
Experiments were performed with 2 images (one grayscale and one color image). The 
table in Figure 63 shows the result for the color image. 

We can see from the results that the higher the y, * ne worst tne image quality. 

20 However, the difference in image quality is not very significant. The pictures look 
very similar and there are no significant changes that can be detected. However as 
predicted, the blocky looking clouds (see Figure 60) are seen when y = 8 or below. 
Therefore by setting y lower we can increase the image quality by some small 
amounts without changing the visual quality of image. Therefore y = 12 was chosen to 

25 be the default value for y parameter in the potential function. 
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Filter Lenp th 

Filter length (F) determines of the size of samples collected for pixel estimations. It is 
obvious that the larger the filter length the longer the processing time. The default 
length given by VM3.0 (B) is 9 pixels. Experiments are performed with varying filter 
5 lengths to examine how the length affects the image quality (MSE and PSNR) and the 
artifact removal ability. Two pictures are used in this experiment (one color and one 
grayscale). The results are shown in the following tables in Figure 64 and 65. 

As we can see from the experimental results, the small the iength, the better the image 
quality and the shorter the processing time. However, artifacts are not completely 
JO removed for length = 5 (or smaller) at high compression ratios. The default value for 
the filter length is chose to be 7 pixels to increase image quality and to decrease 
processing time without degrading the ability for artifact removal. 

Constraint 

The value of constraint (Thl) affects the filter as described in the equations below. 
d - x - x 

15 c(d t ThJ) = sign(d) * Max(0, abs(d) - Max(0, 2*(abs(d) - 7777))) 

From the equations above, we can see that as Thl increases, more pixels will have *d* 
as the output for c(d,ThJ). Therefore, as Thl increases, more estimate value of x will 
be used as the final estimate and the image quality will decrease while the smoothness 
of an image will increase. In order to keep the image quality as high as possible, Thl 
20 should be kept as low as possible. However, artifacts might not be properly removed 
if Thl is set too low. Three images were tested with different level of constraints in 
the experiment. The test results are given in the tables in Figures 66 to 68. 

These results show that image quality decreases as constraint increases. However, 
some artifacts (very small, can only be seen when the picture is enlarged) are not 
25 completely removed when C is equal to 4 or less. In order to achieve the best image 
quality while successfully remove artifacts, the value 8 is suggested to be the default 
value. 
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Iteration 

It is obvious that processing time is directly proportional to the number of iterations. 
However, artifacts might not be removed successfully if the image is not processed 
enough times. Figure 69 clearly shows that image quality decreases as the number of 
5 iteration increases. However, when image is enlarged, small artifacts can be still 
visible for Rl. In order to achieve the best image quality while successfully remove 
artifacts, 2 iterations is suggested to be the default setting. 

Mask Shape 

Different shapes of masks are discussed in Shen's paper, however, only the one with 
10 the shape of a + sign, was implemented in VM3.0 (B). Different masks were 
experimented and the shape of mask is best left unchanged since simplifying the mask 
degrades the performance of the filter and complicating the mask increases processing 
time greatly. 

Using the Modified Post Processing Filter 

15 The adaptive de-ringing is implemented based on the post processing filter in VM3.0 
(B). Few parameters have been eliminated and default values have been established to 
increase the usability of the post processing filter. 

To use the modified post processing filter: 

Usage: posi2 -s width height bpp -i infile -o outfile [-1 mask lower threshold] 
20 [-w mask width] f-t thresh] (-f fjength] J-c constraint! [-r iterations] 

Default values will be used for the parameters if the parameters are not specified in 
the command line. The default parameters are: 

mask lower threshold: lower threshold for edge detection 1 30] 
mask width: width of mask f 12] 
25 thresh: y parameter in the potential function (for estimation) 1 12] 

fjength: filler length 17] 

constraints: constrain Is using in the clipping function [8] 
iterations: number of iterations (2] 

The modified post processing filter performs well with the above default parameters, 

30 however, these parameters can be changed at the command line if the user wishes. 

The post processing filter can now be applied to pictures with 8:1 compression ratio 

73 



SUBSTITUTE SHEET (RULE 26) 



WO 00/49571 



PCT/CAOO/00134 



with very small increase in MSE. The modified post processing filter seems to 
improve image quality for compression ratio beyond 11:1. 

In practice, there are two ways for the user to play with the settings: on the encoder 
side and on the decoder side. 

5 Decoder end 

The decoder will have the full control of the post processing and the of the associate 
parameters. The encoder will have no control on how the images will look at the 
decoder end. Accordingly, there will be no addition to file header since no post 
processing parameters will be included. 

10 Encoder end 

When necessary, the encoder can also predetermine the post processing filter 
parameters. Post processing filter parameters will be stored in the image header, and 
the image will be restored according to these parameters. In this setting, the user at 
the encoding end knows exactly how the image will look at the decoder end. 
15 However, adding these parameters in the header will increase the size of the 
compressed image. 

Total of 6 parameters will need to be packed in the header; mask lower threshold, 
mask width, estimation threshold, filter length, constraints and number of iterations. 
The table in Figure 70suggests range limits on the parameters to reduce the header 
20 size. 

In total, three bytes will be needed to include these parameters in the header. 
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WE CLAIM 

1 . A region-based method for encoding and decoding digital still images to 
produce a scalable, content accessible compressed bit stream comprising the 
steps: 

decomposing and ordering the raw image data into a hierarchy of multi- 
resolution sub-images; 

determining regions of interest; 

defining a region mask to identify regions of interest; 

encoding region masks for regions of interest 

determining region masks for subsequent levels of resolution; and 

scanning and progressively sorting the region data on the basis of the 
magnitude of the multi-resolution coefficients. 

2. An apparatus for the region-based encoding and decoding of digital still 
images that produces a scalable, content accessible compressed bit stream 
comprising: 

a means of decomposing and ordering the raw image data into a hierarchy of 
multi-resolution sub-images; 

means of determining regions of interest; 

means of defining a region mask to identify regions of interest; 

means of encoding region masks for regions of interest; 

means of determining region masks for subsequent levels of resolution; and 

a means for scanning and progressively sorting the region data on the basis of 
the magnitude of the multi-resolution coefficients. 

3. A region-based system for encoding and decoding digital still images that 
produces a scalable, content accessible compressed bit stream and comprises 
the steps: 

decomposing and ordering the raw image data into a hierarchy of multi- 
resolution sub-images; 

determining regions of interest; 

defining a region mask to identify regions of interest; 
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encoding region masks for regions of interest 

determining region masks for subsequent levels of resolution; and 

scanning and progressively sorting the region data on the basis of the 
magnitude of the multi-resolution coefficients. 

4. A method for encoding and decoding digital still images to produce a scalable 
content accessible compressed bit stream comprising the steps: 

decomposing and ordering the raw image data into a hierarchy of multi- 
resolution sub-images; 

setting an initial threshold of significance and creating a significance index; 
determining an initial list of insignificant blocks; 

forming the list of significant coefficients by encoding a significant map using 
a quadtree representation; 

recursively reducing the threshold values and repeating the encoding process 
for each threshold value; and 

transmitting refinement bits of significant coefficients. 

5. An apparatus for encoding and decoding of digital still images that produces a 
scalable, content accessible compressed bit stream comprising: 

a means of decomposing and ordering the raw image data into a hierarchy of 
multi-resolution sub-images; 

means for setting an initial threshold of significance and creating a 
significance index; 

means for determining an initial list of insignificant blocks; 

means of forming the list of significant coefficients by encoding a significant 
map using a quadtree representation; 

a means of recursively reducing the threshold values and repeating the 
encoding process; and 

transmitting refinement bits of significant coefficients. 

6. A method of decoding digital still images to produce a scalable, content 
accessible compressed bit stream comprising the steps: 

decoding the bitstream header; 
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determining the initial threshold values and the array of initial significant 
pixels, insignificant bits and wavelet coefficients; 

decoding the significance maps; 

modifying the significance lists and decoding the refinement bits for each 
threshold level; 

reconstruct the wavelet coefficient array; 

perform the inverse wavelet transform; and 

reconstruct the image. 

7. A method of transmission of digital signals that creates a scalable, content 
accessible bitstream comprising the steps: 

pack the most significant bits of the largest coefficients first followed by 
refinement bits and the most significant bits that are significant for coefficients 
at the next bit level; 

repeat this process in a recursive fashion until the desired compression size is 
obtained; 

calculate the pack ratios to be used for each channel of the wavelet 
decomposition hierarchy by taking the ratio of the two largest amounts of data 
to the smallest amount of data; 

determine the optimal amount of data to allocate for each color channel based 
on the user specified compressed file size; and 

if performing region of interest processing, consider the packing overhead 
introduced by the mask when determining the bit budget for each channel. 
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Data Type: MUXLIST 
Parameters: 

UTotBytcsPacked - long integer total bytes packed into the data buffer for this list 
cScheme - character processing scheme used for data contained in this list 

cHighBit - character highest bit-level where data processing begins for this list 

♦pucMuxBuff - pointer to unsigned character buffer where data for each 

bit-level is packed for this list 

Fields for MUX: information for packing after list processing is complete. 

pliBitPackInfo[ 1 6] - pointer to long integer number of bits packed into the data buffer 

. at each bit-level for this list 
liCurBytesCount - long integer current byte count used for bit budget distribution 

when packing this list 
cCurBitLevel - character current bit-level used for packing this list 

cRemainingBits - character rernaining bits to be packed at a given bit-level when 
data to be packed is not evenly divisible by 8 for this list 
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CALCULATE Channel BiiBudget // determine optimal bit-budget for each color channel. 
INITIALIZE Channel CurrentBitPlane // highest bit plane that exists in each color channel. 
INITIALIZE HCurByiesCount and cRemainingBits FOR each MUX list 

FOR Each Color Channel // process each channel separately. 

WHILE Channel BitBudget > 0 AND Channel CurrentBitPlane >=0 

FOR Each Wavelet Transform level // beginning at lowest resolution level. 

FOR Each Orientation Set of Data // according to lossy case natural processing order. 
IF cCurBitLevei NOT EQUAL to Channel CurrentBitPlane 

CONTINUE 
ELSE 

SET BitLevelBytes to ptiBitPockInfo\ Channel CurrentBitPlane] » 3 

SET RemBits to pliBitPack]nfo\Ouimz\ CurrentBitPlane] & 7 

IF Sum(cRemainingBits, RemBits) >= 8 
INCREMENT BitLevelBytes by 1 
DECREMENT cRemainingBits by 8 - RemBits 

ELSE 

INCREMENT cRemainingBits by RemBits 
END IF 

IF Channel BitBudget >- BitLevelBytes 

INCREMENT UCurBytesCount by BitLevelBytes 
DECREMENT Channel BitBudget by BitLevelBytes 
DECREMENT cCurBitLevei by 1 

ELSE 

INCREMENT UCurBytesCount by Channel BitBudget 
SET Channel BitBudget to 0 
END IF 
END IF 

IF Channel BitBudget EQUALS 0 

BREAK 
ENDIF 
END FOR 
END FOR 

DECREMENT Channel CurBitPlane by 1 
END WHILE 
END FOR 
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Orientation 
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Image Dimension 


Overhead Bits 
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(Rows & Columns) 


(Lossy) 
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(GrayScale) 
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Percentage List Overhead vs. Image 
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Y-channel: R,(LL.-HL.-La,-HH4-HL 3 -LH 3 -HH 3 -HL 2 -LH2-HHr HLrLH,-HH,), 
R2(lX,'HU-LH 4 -HH4-HL3-I^3-HH 3 -HL 2 -LH r HHrHL r LH,-HH 1 ), 
R 3 (LL4-HL4-La,-HH4-HL3-LH 3 -HH 3 -HL 2 -LH 2 -HH 2 - HL,-LH,-HH,), 
R.CU^-HU-LH^-HH^-HLj-^-HHrHU-L^-HHrHLrLHrHH,). 

U-channel: R,(LL 3 - HL 3 -LH r HH 3 -HL2-LH 2 -HH r HLj-LHj-HH,), 
R 2 (LL 3 - HL 3 -LH 3 -HH 3 -HL3-LH r HH 2 - HL,-LHi-HHi), 
R 3 (LL 3 - HL 3 -LH 3 -HH 3 -HL2-LH 2 -HH 2 - HL,-LH,-HH,), 
R,(LL 3 - HL 3 -LH 3 -HH 3 -HL2-LH r HH 2 - HL,-LH,-HH,). 

V-channel: R,(LL 3 - HL 3 -LH 3 -HH 3 -HL2-LH r HH r HL|-LH,-HH,), 
R 2 (LL 3 - HL 3 -LH 3 -HH 3 -HLj-LH 2 -HH 2 - HL,-LHi-HHi), 
R 3 (LL r HL 3 -LH 3 -HH 3 -HL2-LH r HH 2 - HL,-LH r HH,), 
R,(LL3- HL 3 -LH 3 -HH 3 -HL 2 -LH 2 -HH2- HL,-LH r HH,). 
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Ri (Ylw-Yhw-Ylh^-Yhh^-Ulu- Vuj- 

Yhu _ Ylh3-Yhh3 -Uhu-Ulto-Uhh3- Vhu" Vlh3-Vhh3- 
Yhu"Yuo - Yhh2*Uhij*Ulh2-Uhh2*Vhu"Vlh2-Vhh2- 
Yhli -Ylhi - Yhhi -Uhli -Ulhi -Uhhi - Vhli -Vlhi -Vhhi )» 

R2(YlL4-Yhw-YlH4-YhH4-UlL5-Vuj- 

Yhlt Ylhj-Yhh3 -Uhu-Uuo-Uhh3- Vhu- Vlh3-Vhh3- 

Y K L3-Ylh2-YhH3-UhL2-UlH2-UhH2*VhU-VlH2 , 'VhH2* 

Yhli -Ylhi -Yhhi -Uhli -Ulhi -Uhhi - Vhli - Vlhi - Vhhi )» 

R3(Y LL4"YhL<-YlH4-YhH4-UlL3-Vll3- 

YhL3-YlH3-YhH3-UhL3-UlH3-UhH3-VhL3-VlH3-VhH3- 
YhL2-YlH2-YhH2-UhL2-UlH2-UhH2-VhL2-VlH2"VhR2- 

Yhli -Ylhi -Yhhi -Uhli -Ulhi -Uhhi - Vhli -Vlhi - Vhhi )» 

R-«(Y LL4-YhL4-YlH4-YhH4-UlLJ-VlL3- 

YhL3-YlH3" YhH3-UhL3-UlH3-UhH3-Vhl3 -VlH3- VhH3* 

Yhij-Yuc-Yhki'Uhij-Ulhj-Uhh^-Vhlj-Vljc-Vhhi- 
Yhli -Ylhi - Yhh 1 -Uhli -Ulhi -Uhhi - Vhli - v lh i - Vhhi ) • 



FIGURE 40 



18/35 



SUBSTITUTE SHEET (RULE 26) 



WO 00/49571 



PCT/CAOO/00134 



Yhu-Uhlj-Vhu-Ylhj-Uuo-Vuo-Yhhb-Uhhs-Vhhs- 
YHU-Ui^-V^-Yuc-Uuc-Vuo-YHHj-UHm-VHH^ 
Yhli -Uhli -Vhli -Ylh, -Ulhi -Vu,, -Yhhi -Uhhi - v hhi ), 
RiCYlu-Yhw-Yum-Yhh^-Uuj-Vuj- 

YhL3*Uhjj-VhL3-Ylh3*UlH3-Vlh3-YhH3"UhH3"VhiO" 
YKLa-UHLS-VHu-YLHj-Uuo-Vuc-Yioc-UHffi-Vjoc- 

Yhli -Uhli - Vhli -Ylh, -Ulhi - Vmi -Yhhi -Uhhi - Vhhi), 
R3CYLL1 -Yhw- YlH4- YhH4-Ull3- Vuj- 

Yh1J"Uhij*Vhl3"YlH3~UlH3~Vlh3"Yhh3-UhH3'VhH3- 
YHU-Ui^-VHLS-Yuc-UiJC-ViJc-YHro-UHHS-VHm- 

Yhli -Uhli -Vhlj -Ylh) -Ulhi - Vlh, -Yhhi -Uhhi -Vhhi ), 
^(Yli^-Yhw-Ylh^-Yhh^-Ulu-Vuj- 

YHL3-UHL3-VHU-YLH3-ULH3-VLH3-YHH3-UHH3-VHH3- 
YhU-UhL2-Vhl2-YlH2-UlH2-VlH2-YhH2-UhH2-VhH2- 

Yhli -Uhli -Vhlj -Ylhi -Ulhi - Vlhi -Yhhi -Uhhi - Vhhi )• 
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Yhl3-Ylh3-Yh}m-Uhl4-Ulh4-Uhh4-Vhl3-Vlh3-Vhh3- 

yHL2-YLH2-YHH2-UnL2-ULH2-UHH2-VHU-VLH2-VnH2- 

y hlj -Ylhi -Yhhi -Uhli -Ulhi -Uhhi -Vhli -Vlhi -Vhhi ), 

"Vix4-Yhl4-Ylh4 - Yhh4-Uhi^-Uiji4-Uhh4-Vhi^-Vlh4-Vhh4- 

YHL3-YLH3-YHH4-UHW-ULH4-UHH4-VHU-VLH3-VHH3- 

yhu-ylh2-yhh2-uhl2-ulh2-uhh2-vhl2-vlh2-vhh2- 
Yhlj -Ylhi -Yhhi -Uhli -Ulhi -Uhhi -Vhli -Vlhi -Vhhi ), 
WOWUl^-Vlw-Yhu-Ylw-Yh^^ 

YhL3-Ylh3-Yhh4-UhL4-UlH4-UhH4-Vhl3-VlH3-Vhh3- 
YhL2-Ylh2-YhH2-Uhl2-Ulh2-UhH2-VhL2-VlH2-Vhh2- 

Yhli "Ylhi -Yhhi -Uhli -Ulhi -Uhhi -Vhli - Vlhi - Vhhi ). 
R4(Yix,-U u ,,-V u ^-Y KL ,-Y 1 ^-Y }ffl ^U m .,.U L H4-UHH.-V H ^.V L „,-V fiH4 - 
Yhl3-Ylh3-Yhh4-Uhl4-Ulh4-Uhh4-Vhl3-Vlh3-Vhh3- 

Yhl2-Ylh2-YhH2-Uhl2-UlH2-UhH2-Vhl2-VlH2-VhH2- 

Yhli -Ylh, -Yhhi -Uhli -Ulhi -Uhhi -Vhli -Vlhi-Vhhi ). 
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YlL4*YhL4-YlH4-YhB4-UlL3-Vll3- 

VhU-YlH3-YhH3-Uhl3-UlH3-UhH3-VhL3-Vlh3-VhH3- 
YhL2-YlH2-Yhh2-UhL3-UlH2-UhH2-Vhl2-VlH2-VhH2* 

Yhli -Ylhi -Yhhi -Uhli -Ulhi -Uhhi -Vhli -Vlhi -Vhhi (Rl, R2, R3 , R4). 
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Y L u-Ulu-V lw -Y^-Y L h4-Y^-Uhl4-U l „ 4 -U wm -Vhl4-V LH 4-V„ H4 - 
Yhl3-Y L h3-Yhh4-Uhu-Ulh4-Uhh4-Vhl3-Vlh3- v hh3- 

YHL2-Y L H2-YHH2-UHL2-ULH2-UHH2-VHL2-VLH:-VHHr 

VHLrYLHrYHHi-UHLj-ULHrUHHrVHLi-VLHrVra, (Rl, R2, R3, R4) 
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MSE vs. Bpp for Glacier Park Image 
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MSE vi. Bpp for Glacier Parle 
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MSE vs. Bpp for Glacier Park Image 
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MSE vs. Region Data Percentages for Glacier Park Image 
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1 Lossless Case 


backed Item 


bits 


Selections / Comments 


header size 


s 


Total Header is normally about 6 bytes 


o oil grayscale / 2A oil color 


1 




image width 


10 


(64-1024 columns) 


Entropy Coding 


2 


None, Basic, Adaptive 


image Height 


10 


(64-1024 rows) 


Wavelet Transform Levels 


3 


Level >s 1 


Region Channels 


2 


1.2, 3,4 


Mask Type 


2 


None, User, Auto (DCT) 


Pack Raw Mask Flap 


1 


Read in file for mask 


Mask Procedure 


3 


DCT Common Mask, Raw Common Mask with VM Translation, 


Down Sampling Type 


2 


Heuristic DCT Down Sampling 


Region Start Level 


4 


No Regions to # wavelet levels 


Lossy Flag 


1 


Lossless 


Color Transform Type 


2 


2 Internal. Y1Q (full data sets) 


Sort Type 


1 


1D.EQW 


Wavelet Kernel 


2 


Lifting Scheme 
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Lossv Case 
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